bug#54970: 28.1; Some emoji no longer display

bug#54970: 28.1; Some emoji no longer display

[Howard Melman]

(I normally use the macport but I was able to reproduce this on vanilla
emacs 28.1 from emacsformacosx so please forgive if I get some details
wrong.)

Using emacs -Q on MacOS 11.6.5:

Use C-x 8 RET and select FORK AND KINFE WITH PLATE

I don't actually see the character on vanilla emacs but if I put point
before it the cursor displays a bit wider if that makes sense.  On the
macport I see tofu.

C-u C-x = with point just before the character shows:

================

             position: 319 of 319 (100%), column: 0
            character: 🍽 (displayed as 🍽) (codepoint 127869, #o371575, #x1f37d)
              charset: unicode (Unicode (ISO10646))
code point in charset: 0x1F37D
               script: symbol
               syntax: w 	which means: word
             category: .:Base
             to input: type "C-x 8 RET 1f37d" or "C-x 8 RET FORK AND KNIFE WITH PLATE"
          buffer code: #xF0 #x9F #x8D #xBD
            file code: #xF0 #x9F #x8D #xBD (encoded by coding system utf-8-unix)
              display: no font available

Character code properties: customize what to show
  name: FORK AND KNIFE WITH PLATE
  general-category: So (Symbol, Other)
  decomposition: (127869) ('🍽')

There are text properties here:
  fontified            nil

================

Following the instructions in NEWS I did:

(set-fontset-font t 'emoji
    '("Apple Color Emoji" . "iso10646-1") nil 'prepend)

But nothing changed.  Doing this displayed the emoji character:

(set-fontset-font t 'symbol
    '("Apple Color Emoji" . "iso10646-1") nil 'prepend)

I gather the difference is that this character is part of script symbol
and not emoji.  I don't understand the difference and don't know if
emacs is wrong about this or if I should know the difference. In other
places on macOS and iOS it's just an emoji like any other.  I do know
that the instructions in NEWS didn't help me display a character that
displayed fine for me in the macport of Emacs 27.2.

I suspect other "emoji" have the same problem.  On the macport with
vertico and marginalia enabled (so I see completion candidates and
their character displays) I feel that I see tofu for more emojis than 
Emacs 27 displayed (e.g, 0x1F6CF which is BED).

Howard



In GNU Emacs 28.1 (build 1, x86_64-apple-darwin18.7.0, NS appkit-1671.60 Version 10.14.6 (Build 18G95))
of 2022-04-04 built on builder10-14.lan
Windowing system distributor 'Apple', version 10.3.2022
System Description:  macOS 11.6.5

Configured using:
'configure --with-ns '--enable-locallisppath=/Library/Application
Support/Emacs/${version}/site-lisp:/Library/Application
Support/Emacs/site-lisp' --with-modules'

Configured features:
ACL GMP GNUTLS JSON LIBXML2 MODULES NOTIFY KQUEUE NS PDUMPER THREADS
TOOLKIT_SCROLL_BARS ZLIB

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny dired dired-loaddefs
rfc822 mml mml-sec epa derived epg rfc6068 epg-config gnus-util rmail
rmail-loaddefs auth-source cl-seq eieio eieio-core cl-macs
eieio-loaddefs password-cache json map text-property-search time-date
subr-x seq byte-opt gv bytecomp byte-compile cconv mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader cl-loaddefs
cl-lib sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils
iso-transl tooltip eldoc paren electric uniquify ediff-hook vc-hooks
lisp-float-type elisp-mode mwheel term/ns-win ns-win ucs-normalize
mule-util term/common-win tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode lisp-mode prog-mode register
page tab-bar menu-bar rfn-eshadow isearch easymenu timer select
scroll-bar mouse jit-lock font-lock syntax font-core term/tty-colors
frame minibuffer cl-generic cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932
hebrew greek romanian slovak czech european ethiopic indian cyrillic
chinese composite emoji-zwj charscript charprop case-table epa-hook
jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice button
loaddefs faces cus-face macroexp files window text-properties overlay
sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote threads kqueue cocoa ns multi-tty
make-network-process emacs)

Memory information:
((conses 16 50130 8613)
(symbols 48 6722 2)
(strings 32 18540 1579)
(string-bytes 1 611343)
(vectors 16 15159)
(vector-slots 8 455175 12837)
(floats 8 21 40)
(intervals 56 209 0)
(buffers 992 11))

-- 

Howard

bug#54970: 28.1; Some emoji no longer display

[Lars Ingebrigtsen]

Howard Melman <hmelman@gmail.com> writes:

> (I normally use the macport but I was able to reproduce this on vanilla
> emacs 28.1 from emacsformacosx so please forgive if I get some details
> wrong.)
>
> Using emacs -Q on MacOS 11.6.5:
>
> Use C-x 8 RET and select FORK AND KINFE WITH PLATE
>
> I don't actually see the character on vanilla emacs but if I put point
> before it the cursor displays a bit wider if that makes sense.  On the
> macport I see tofu.

I'm unable to reproduce this on Debian with emacs -Q in 28.1, so I guess
it depends on the font selection code?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#54970: 28.1; Some emoji no longer display

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> I'm unable to reproduce this on Debian with emacs -Q in 28.1, so I guess
> it depends on the font selection code?

But I can reproduce it on Macos.  I forget whether this is supposed to
work out of the box, or whether you have to set the fontset things on
Macos.  Alan probably knows; I've added him to the CCs.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#54970: 28.1; Some emoji no longer display

[Howard Melman]

On Apr 16, 2022, at 10:55 AM, Lars Ingebrigtsen <larsi@gnus.org> wrote:
> 
> Lars Ingebrigtsen <larsi@gnus.org> writes:
> 
>> I'm unable to reproduce this on Debian with emacs -Q in 28.1, so I guess
>> it depends on the font selection code?
> 
> But I can reproduce it on Macos.  I forget whether this is supposed to
> work out of the box, or whether you have to set the fontset things on
> Macos.  Alan probably knows; I've added him to the CCs.

For me in vanilla emacs -Q other emojis like 0x1f468 "MAN" and 
0x1f415 "DOG" display out of the box in Apple Color Emoji.  
So I would think this should too.

Howard

bug#54970: 28.1; Some emoji no longer display

[Alan Third]

On Sat, Apr 16, 2022 at 11:27:52AM -0400, Howard Melman wrote:
> On Apr 16, 2022, at 10:55 AM, Lars Ingebrigtsen <larsi@gnus.org> wrote:
> > 
> > Lars Ingebrigtsen <larsi@gnus.org> writes:
> > 
> >> I'm unable to reproduce this on Debian with emacs -Q in 28.1, so I guess
> >> it depends on the font selection code?
> > 
> > But I can reproduce it on Macos.  I forget whether this is supposed to
> > work out of the box, or whether you have to set the fontset things on
> > Macos.  Alan probably knows; I've added him to the CCs.
> 
> For me in vanilla emacs -Q other emojis like 0x1f468 "MAN" and 
> 0x1f415 "DOG" display out of the box in Apple Color Emoji.  
> So I would think this should too.

I think it's somewhat inconsistent if you don't set the font:

    (set-fontset-font t 'symbol "Apple Color Emoji")

Please let us know if that makes any difference.
-- 
Alan Third

bug#54970: 28.1; Some emoji no longer display

[Howard Melman]

On Apr 16, 2022, at 12:02 PM, Alan Third <alan@idiocy.org> wrote:
> 
> On Sat, Apr 16, 2022 at 11:27:52AM -0400, Howard Melman wrote:
>> On Apr 16, 2022, at 10:55 AM, Lars Ingebrigtsen <larsi@gnus.org> wrote:
>>> 
>>> Lars Ingebrigtsen <larsi@gnus.org> writes:
>>> 
>>>> I'm unable to reproduce this on Debian with emacs -Q in 28.1, so I guess
>>>> it depends on the font selection code?
>>> 
>>> But I can reproduce it on Macos.  I forget whether this is supposed to
>>> work out of the box, or whether you have to set the fontset things on
>>> Macos.  Alan probably knows; I've added him to the CCs.
>> 
>> For me in vanilla emacs -Q other emojis like 0x1f468 "MAN" and 
>> 0x1f415 "DOG" display out of the box in Apple Color Emoji.  
>> So I would think this should too.
> 
> I think it's somewhat inconsistent if you don't set the font:
> 
>    (set-fontset-font t 'symbol "Apple Color Emoji")
> 
> Please let us know if that makes any difference.

As I said in my initial report, that (or something similar) did solve it for me.

But the NEWS report said to do this:

    (set-fontset-font t 'emoji
                      '("My New Emoji Font" . "iso10646-1") nil 'prepend)

Which did not work for me, but this did:

    (set-fontset-font t 'symbol
                     '("Apple Color Emoji" . "iso10646-1") nil 'prepend)

I'm not clear about what the iso10646-1 part does or if it's needed.  
And the NEWS entry says:

    The Emoji characters are now assigned to a special script, 'emoji', so
    as to make it easier to customize fontsets for Emoji display, as in
    the above example.  (Previously, the Emoji characters were assigned to
    the 'symbol' script, together with other symbol and punctuation
    characters.)

Which reads to me as conflicting with needing to set the emoji font for 'symbol.

Howard

bug#54970: 28.1; Some emoji no longer display

[Eli Zaretskii]

> From: Howard Melman <hmelman@gmail.com>
> Date: Sat, 16 Apr 2022 12:21:20 -0400
> Cc: Lars Ingebrigtsen <larsi@gnus.org>, 54970@debbugs.gnu.org
> 
> >    (set-fontset-font t 'symbol "Apple Color Emoji")
> > 
> > Please let us know if that makes any difference.
> 
> As I said in my initial report, that (or something similar) did solve it for me.
> 
> But the NEWS report said to do this:
> 
>     (set-fontset-font t 'emoji
>                       '("My New Emoji Font" . "iso10646-1") nil 'prepend)
> 
> Which did not work for me, but this did:
> 
>     (set-fontset-font t 'symbol
>                      '("Apple Color Emoji" . "iso10646-1") nil 'prepend)

That's because we don't want to advertise the (non-free) Apple font.

> I'm not clear about what the iso10646-1 part does or if it's needed.  

It makes the setting work more reliably in this case.

bug#54970: 28.1; Some emoji no longer display

[Howard Melman]

> On Apr 16, 2022, at 12:42 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Howard Melman <hmelman@gmail.com>
>> Date: Sat, 16 Apr 2022 12:21:20 -0400
>> Cc: Lars Ingebrigtsen <larsi@gnus.org>, 54970@debbugs.gnu.org
>> 
>>>   (set-fontset-font t 'symbol "Apple Color Emoji")
>>> 
>>> Please let us know if that makes any difference.
>> 
>> As I said in my initial report, that (or something similar) did solve it for me.
>> 
>> But the NEWS report said to do this:
>> 
>>    (set-fontset-font t 'emoji
>>                      '("My New Emoji Font" . "iso10646-1") nil 'prepend)
>> 
>> Which did not work for me, but this did:
>> 
>>    (set-fontset-font t 'symbol
>>                     '("Apple Color Emoji" . "iso10646-1") nil 'prepend)
> 
> That's because we don't want to advertise the (non-free) Apple font.

That's fine, but the first is for the script symbol 'emoji and the second
is for the script symbol 'symbol and using the first with the Apple font
didn't work.

>> I'm not clear about what the iso10646-1 part does or if it's needed.  
> 
> It makes the setting work more reliably in this case.

Not in my experience :) What I'm not clear about is why the registry
"iso10646-1" is needed if I'm specifying a script for emoji which I think
is always unicode.

Howard

bug#54970: 28.1; Some emoji no longer display

[Eli Zaretskii]

> From: Howard Melman <hmelman@gmail.com>
> Date: Sat, 16 Apr 2022 13:17:45 -0400
> Cc: Alan Third <alan@idiocy.org>,
>  Lars Ingebrigtsen <larsi@gnus.org>,
>  54970@debbugs.gnu.org
> 
> 
> 
> > On Apr 16, 2022, at 12:42 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> > 
> >> From: Howard Melman <hmelman@gmail.com>
> >> Date: Sat, 16 Apr 2022 12:21:20 -0400
> >> Cc: Lars Ingebrigtsen <larsi@gnus.org>, 54970@debbugs.gnu.org
> >> 
> >>>   (set-fontset-font t 'symbol "Apple Color Emoji")
> >>> 
> >>> Please let us know if that makes any difference.
> >> 
> >> As I said in my initial report, that (or something similar) did solve it for me.
> >> 
> >> But the NEWS report said to do this:
> >> 
> >>    (set-fontset-font t 'emoji
> >>                      '("My New Emoji Font" . "iso10646-1") nil 'prepend)
> >> 
> >> Which did not work for me, but this did:
> >> 
> >>    (set-fontset-font t 'symbol
> >>                     '("Apple Color Emoji" . "iso10646-1") nil 'prepend)
> > 
> > That's because we don't want to advertise the (non-free) Apple font.
> 
> That's fine, but the first is for the script symbol 'emoji and the second
> is for the script symbol 'symbol and using the first with the Apple font
> didn't work.

Which is how it should be, AFAIU.  That character is not in the
Unicode Emoji sequences database.

> >> I'm not clear about what the iso10646-1 part does or if it's needed.  
> > 
> > It makes the setting work more reliably in this case.
> 
> Not in my experience :) What I'm not clear about is why the registry
> "iso10646-1" is needed if I'm specifying a script for emoji which I think
> is always unicode.

Nobody knows.  You are welcome to read the font-related code in Emacs,
and if you figure that out, please be sure to tell us.

bug#54970: 28.1; Some emoji no longer display

[Howard Melman]

> On Apr 16, 2022, at 2:26 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> That character is not in the Unicode Emoji sequences database.

It's not?  I don't know anything about emojis in Unicode other than
it's more complicated than anyone would expect, but I see it in:
http://git.savannah.gnu.org/cgit/emacs.git/tree/admin/unidata/emoji-sequences.txt?h=emacs-28

1F37D FE0F    ; Basic_Emoji                  ; fork and knife with plate                                      # E0.7   [1] (🍽️)

Or am I misunderstanding something?

Howard

bug#54970: 28.1; Some emoji no longer display

[Eli Zaretskii]

> From: Howard Melman <hmelman@gmail.com>
> Date: Sat, 16 Apr 2022 15:19:25 -0400
> Cc: alan@idiocy.org,
>  larsi@gnus.org,
>  54970@debbugs.gnu.org
> 
> 
> > On Apr 16, 2022, at 2:26 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> > 
> >> That character is not in the Unicode Emoji sequences database.
> 
> It's not?  I don't know anything about emojis in Unicode other than
> it's more complicated than anyone would expect, but I see it in:
> http://git.savannah.gnu.org/cgit/emacs.git/tree/admin/unidata/emoji-sequences.txt?h=emacs-28
> 
> 1F37D FE0F    ; Basic_Emoji                  ; fork and knife with plate                                      # E0.7   [1] (🍽️)
> 
> Or am I misunderstanding something?

Your recipe didn't mention the U+FE0F codepoint.  You only talked
about the lone U+1F37D.  When followed by U+FE0F, Emacs indeed ought
to display the sequence as Emoji, and that should not require you to
change the font used for the 'symbol' pseudo-script.  Are you saying
that isn't happening in your case?

bug#54970: 28.1; Some emoji no longer display

[Howard Melman]

> On Apr 16, 2022, at 3:22 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> 
>> It's not?  I don't know anything about emojis in Unicode other than
>> it's more complicated than anyone would expect, but I see it in:
>> http://git.savannah.gnu.org/cgit/emacs.git/tree/admin/unidata/emoji-sequences.txt?h=emacs-28
>> 
>> 1F37D FE0F    ; Basic_Emoji                  ; fork and knife with plate                                      # E0.7   [1] (🍽️)
>> 
>> Or am I misunderstanding something?
> 
> Your recipe didn't mention the U+FE0F codepoint.  You only talked
> about the lone U+1F37D.  When followed by U+FE0F, Emacs indeed ought
> to display the sequence as Emoji, and that should not require you to
> change the font used for the 'symbol' pseudo-script.  Are you saying
> that isn't happening in your case?

Indeed in Emacs 28 -Q if I

C-x 8 RET FORK AND KNIFE WITH PLATE RET
C-x 8 RET FE0F RET

I see the emoji displayed.

If I use the system emoji selector to insert a character into emacs
It only puts in "the lone U+1F37D" (at least in emacs where I can
check). In other apps this displays as an emoji, in emacs it's a blank.

I said, I don't understand this stuff.  Is this extra codepoint supposed
to be added for me?  It doesn't seem like other apps require it.

I know this, in Emacs 27 macport I used C-x 8 RET and searched for plate
and entered this character into a file and it displayed as an emoji.
Several others I entered this way displayed fine too.  I upgraded to
Emacs 28 and the other characters display fine and this one didn't.  
I followed the instructions in NEWS and they didn't help.

If the answer is just, this kind of emoji is different (though nothing else
really exposes to it me as different), then I think the instructions (in NEWS
or wherever) should just tell me to 

(set-fontset-font t 'symbol '("My New Emoji Font" . "iso10646-1") nil 'prepend)

and not

(set-fontset-font t 'emoji '("My New Emoji Font" . "iso10646-1") nil 'prepend)

But I'd really like to understand the difference.

Howard

bug#54970: 28.1; Some emoji no longer display

[Eli Zaretskii]

> From: Howard Melman <hmelman@gmail.com>
> Date: Sat, 16 Apr 2022 16:07:38 -0400
> Cc: alan@idiocy.org,
>  larsi@gnus.org,
>  54970@debbugs.gnu.org
> 
> > Your recipe didn't mention the U+FE0F codepoint.  You only talked
> > about the lone U+1F37D.  When followed by U+FE0F, Emacs indeed ought
> > to display the sequence as Emoji, and that should not require you to
> > change the font used for the 'symbol' pseudo-script.  Are you saying
> > that isn't happening in your case?
> 
> Indeed in Emacs 28 -Q if I
> 
> C-x 8 RET FORK AND KNIFE WITH PLATE RET
> C-x 8 RET FE0F RET
> 
> I see the emoji displayed.
> 
> If I use the system emoji selector to insert a character into emacs
> It only puts in "the lone U+1F37D" (at least in emacs where I can
> check). In other apps this displays as an emoji, in emacs it's a blank.

"Blank" meaning what?

Emacs should display the character if there's a font on the system
available to Emacs that has a glyph for that codepoint.  It just might
not display it as an Emoji, but as a slightly different symbol, and
usually with a different font and without the colors inherent in Emoji
display.  That's what happens on my system.  If that character doesn't
display at all for you, maybe your fonts don't support it?

The fundamental issue here is that some symbols have both an Emoji
presentation and a "text" presentation, and the VARIATION SELECTOR n
characters tell the rendering system which presentation to display.
The U+FE0F VARIATION SELECTOR 16 character tells Emacs to show the
Emoji presentation.  Without the variation selectors, Emacs displays
as Emoji only characters that are explicitly given the Emoji
presentation as the default by the Unicode Character Database files.
And AFAICT, U+1F37D is not one of them.

(Robert, please correct me if I'm wrong here.)

> I said, I don't understand this stuff.  Is this extra codepoint supposed
> to be added for me?  It doesn't seem like other apps require it.

Emacs currently doesn't insert the variation selectors automatically,
although perhaps the Emoji input method should (or maybe already
does, I didn't check).

> I know this, in Emacs 27 macport I used C-x 8 RET and searched for plate
> and entered this character into a file and it displayed as an emoji.
> Several others I entered this way displayed fine too.  I upgraded to
> Emacs 28 and the other characters display fine and this one didn't.  
> I followed the instructions in NEWS and they didn't help.

Macport Emacs is a separate project that uses its own policies in
these somewhat gray areas.

> If the answer is just, this kind of emoji is different (though nothing else
> really exposes to it me as different), then I think the instructions (in NEWS
> or wherever) should just tell me to 
> 
> (set-fontset-font t 'symbol '("My New Emoji Font" . "iso10646-1") nil 'prepend)
> 
> and not
> 
> (set-fontset-font t 'emoji '("My New Emoji Font" . "iso10646-1") nil 'prepend)

No, that wouldn't be right.  We introduced the special 'emoji'
pseudo-script in Emacs 28 to solve the problems of being unable to
distinguish between Emoji codepoints and the other symbols and
punctuation.  We definitely do NOT want to use an Emoji font for
symbols and punctuation characters and character sequences that aren't
Emoji.  Moreover, configuring the fontset to use some font for the
'symbol' pseudo-script by default doesn't do what you expect, because
Emacs uses the default font for symbols for which the default font has
a glyph.

So I think the recipe in NEWS is correct, and your expectations were
inconsistent with the Emacs support for Emoji, at least with its state
in Emacs 28.1.

bug#54970: 28.1; Some emoji no longer display

[Robert Pluim]

>>>>> On Sun, 17 Apr 2022 08:53:37 +0300, Eli Zaretskii <eliz@gnu.org> said:

    Eli> The fundamental issue here is that some symbols have both an Emoji
    Eli> presentation and a "text" presentation, and the VARIATION SELECTOR n
    Eli> characters tell the rendering system which presentation to display.
    Eli> The U+FE0F VARIATION SELECTOR 16 character tells Emacs to show the
    Eli> Emoji presentation.  Without the variation selectors, Emacs displays
    Eli> as Emoji only characters that are explicitly given the Emoji
    Eli> presentation as the default by the Unicode Character Database files.
    Eli> And AFAICT, U+1F37D is not one of them.

    Eli> (Robert, please correct me if I'm wrong here.)

Thatʼs exactly how it works. See
<https://unicode.org/emoji/charts/emoji-style.html> for a listing of
the affected codepoints and default styles.

    >> I said, I don't understand this stuff.  Is this extra codepoint supposed
    >> to be added for me?  It doesn't seem like other apps require it.

    Eli> Emacs currently doesn't insert the variation selectors automatically,
    Eli> although perhaps the Emoji input method should (or maybe already
    Eli> does, I didn't check).

The stuff Lars added on master puts in variation selectors where
needed.

    Eli> No, that wouldn't be right.  We introduced the special 'emoji'
    Eli> pseudo-script in Emacs 28 to solve the problems of being unable to
    Eli> distinguish between Emoji codepoints and the other symbols and
    Eli> punctuation.  We definitely do NOT want to use an Emoji font for
    Eli> symbols and punctuation characters and character sequences that aren't
    Eli> Emoji.  Moreover, configuring the fontset to use some font for the
    Eli> 'symbol' pseudo-script by default doesn't do what you expect, because
    Eli> Emacs uses the default font for symbols for which the default font has
    Eli> a glyph.

Modulo `use-default-font-for-symbols'

    Eli> So I think the recipe in NEWS is correct, and your expectations were
    Eli> inconsistent with the Emacs support for Emoji, at least with its state
    Eli> in Emacs 28.1.

Iʼm not sure what we could change. I guess we could add a
configuration variable that says 'treat every code point that has a
default text presentation and an emoji one as emoji', except we
already have that: VARIATION SELECTOR 16

Robert
-- 

bug#54970: 28.1; Some emoji no longer display

[Howard Melman]

[-- Attachment #1: Type: text/plain, Size: 1893 bytes --]



> On Apr 17, 2022, at 1:53 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> Indeed in Emacs 28 -Q if I
>> 
>> C-x 8 RET FORK AND KNIFE WITH PLATE RET
>> C-x 8 RET FE0F RET
>> 
>> I see the emoji displayed.
>> 
>> If I use the system emoji selector to insert a character into emacs
>> It only puts in "the lone U+1F37D" (at least in emacs where I can
>> check). In other apps this displays as an emoji, in emacs it's a blank.
> 
> "Blank" meaning what?
> 

> Emacs should display the character if there's a font on the system
> available to Emacs that has a glyph for that codepoint.  It just might
> not display it as an Emoji, but as a slightly different symbol, and
> usually with a different font and without the colors inherent in Emoji
> display.  That's what happens on my system.  If that character doesn't
> display at all for you, maybe your fonts don't support it?

Blank as I described in my original report where I included the output
of C-u C-x = which does say there's no font available.  Following the 
recipe with point before the added character it looks like this:



with point after the character it looks like this:



On the macport I see tofu which I think is better.

> Macport Emacs is a separate project that uses its own policies in
> these somewhat gray areas.

I realize.  I originally report this there because it did previously handle
emoji better on mac but as of Emacs 28, it's NEWS says:

    Emoji composition handling is aligned with upstream.  You may find
    some incompatibilities with the previous versions.

In email, YAMAMOTO Mitsuharu said to me about this:
"I only mentioned composition, but font selection is also affected."

When I confirmed it was not displayed by the NS port either I reported
it here.

So FYI, I think in this area the macport policies are trying to
converge with Emacs' policies, though I don't want to speak for them.

Howard

[-- Attachment #2.1: Type: text/html, Size: 3559 bytes --]

[-- Attachment #2.2: Screen Shot 2022-04-17 at 9.34.55 AM.png --]
[-- Type: image/png, Size: 11899 bytes --]

[-- Attachment #2.3: Screen Shot 2022-04-17 at 9.37.07 AM.png --]
[-- Type: image/png, Size: 7518 bytes --]

bug#54970: 28.1; Some emoji no longer display

[Howard Melman]

> On Apr 17, 2022, at 9:34 AM, Robert Pluim <rpluim@gmail.com> wrote:
> 
>>>>>> On Sun, 17 Apr 2022 08:53:37 +0300, Eli Zaretskii <eliz@gnu.org> said:
> 
>    Eli> The fundamental issue here is that some symbols have both an Emoji
>    Eli> presentation and a "text" presentation, and the VARIATION SELECTOR n
>    Eli> characters tell the rendering system which presentation to display.
>    Eli> The U+FE0F VARIATION SELECTOR 16 character tells Emacs to show the
>    Eli> Emoji presentation.  Without the variation selectors, Emacs displays
>    Eli> as Emoji only characters that are explicitly given the Emoji
>    Eli> presentation as the default by the Unicode Character Database files.
>    Eli> And AFAICT, U+1F37D is not one of them.
> 
>    Eli> (Robert, please correct me if I'm wrong here.)
> 
> Thatʼs exactly how it works. See
> <https://unicode.org/emoji/charts/emoji-style.html> for a listing of
> the affected codepoints and default styles.

Thanks for all this info.  So on that page, in the second headed section of the 
table "Emoji Font" is where U+1F37D appears.  In the "text-vs" row, which
I think is the case of a lone U+1F37D, I see the emoji glyph in my mac browser.
The description in that header says:

  “text+ts” should be monochrome; everything else should be colorful & monospace.

which matches what I see.  So I think, a lone U+1F37D should be displayed
as an "emoji glyph".  

Can emacs be configured to display these lone codepoints via my emoji font?
I gather that's what using the 'symbol script does but also includes more.
Can I (or emacs out-of-the-box) be more selective in the call to 
set-fontset-font or some other api?

>>> I said, I don't understand this stuff.  Is this extra codepoint supposed
>>> to be added for me?  It doesn't seem like other apps require it.
> 
>    Eli> Emacs currently doesn't insert the variation selectors automatically,
>    Eli> although perhaps the Emoji input method should (or maybe already
>    Eli> does, I didn't check).
> 
> The stuff Lars added on master puts in variation selectors where
> needed.

The emoji input method isn't on 28 so I can't check, but FWIW this seems
to not match the behavior I see using the mac system emoji picker
which seems to just insert a lone U+1F37D when I pick this emoji.

And I'll add, if that's displayed equivalently I'd prefer it, because I wouldn't
have to deal with "extra invisible characters" after the glyph when
using emacs editing commands (unless this is different behavior in 29
than in 28 when I add the variation selector character).

>    Eli> No, that wouldn't be right.  We introduced the special 'emoji'
>    Eli> pseudo-script in Emacs 28 to solve the problems of being unable to
>    Eli> distinguish between Emoji codepoints and the other symbols and
>    Eli> punctuation.  We definitely do NOT want to use an Emoji font for
>    Eli> symbols and punctuation characters and character sequences that aren't
>    Eli> Emoji.  Moreover, configuring the fontset to use some font for the
>    Eli> 'symbol' pseudo-script by default doesn't do what you expect, because
>    Eli> Emacs uses the default font for symbols for which the default font has
>    Eli> a glyph.
> 
> Modulo `use-default-font-for-symbols'

FWIW this variable set to t for me which I think is the default.

>    Eli> So I think the recipe in NEWS is correct, and your expectations were
>    Eli> inconsistent with the Emacs support for Emoji, at least with its state
>    Eli> in Emacs 28.1.
> 
> Iʼm not sure what we could change. I guess we could add a
> configuration variable that says 'treat every code point that has a
> default text presentation and an emoji one as emoji', except we
> already have that: VARIATION SELECTOR 16

I think the section I mentioned above is this case, that things in the
"emojiFont" grouping, w/o a variation selector should be presented
colorful and monospace (which I take to mean "as emoji").  Am I
misunderstanding?

Howard

bug#54970: 28.1; Some emoji no longer display

[Eli Zaretskii]

> From: Howard Melman <hmelman@gmail.com>
> Date: Sun, 17 Apr 2022 10:35:11 -0400
> Cc: Robert Pluim <rpluim@gmail.com>,
>  Alan Third <alan@idiocy.org>,
>  Lars Ingebrigtsen <larsi@gnus.org>,
>  54970@debbugs.gnu.org
> 
> > "Blank" meaning what?
> > 
> 
> > Emacs should display the character if there's a font on the system
> > available to Emacs that has a glyph for that codepoint.  It just might
> > not display it as an Emoji, but as a slightly different symbol, and
> > usually with a different font and without the colors inherent in Emoji
> > display.  That's what happens on my system.  If that character doesn't
> > display at all for you, maybe your fonts don't support it?
> 
> Blank as I described in my original report where I included the output
> of C-u C-x = which does say there's no font available.  Following the 
> recipe with point before the added character it looks like this:
> 
> 
> 
> with point after the character it looks like this:
> 
> 
> 
> On the macport I see tofu which I think is better.

Emacs also displays tofu if it finds no suitable font.  What you
describe sounds like Emacs found some font that claimed to have a
glyph for this codepoint, but the glyph displays as blank.  Which is,
of course, in contradiction with the "no font available" part, so I
admit I don't understand what happened on your system.  Was that in
"emacs -Q"?

bug#54970: 28.1; Some emoji no longer display

[Howard Melman]

> On Apr 17, 2022, at 10:44 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> Emacs also displays tofu if it finds no suitable font.  What you
> describe sounds like Emacs found some font that claimed to have a
> glyph for this codepoint, but the glyph displays as blank.  Which is,
> of course, in contradiction with the "no font available" part, so I
> admit I don't understand what happened on your system.  Was that in
> "emacs -Q"?

Yes it was.

Howard

bug#54970: 28.1; Some emoji no longer display

[Robert Pluim]

>>>>> On Sun, 17 Apr 2022 10:35:17 -0400, Howard Melman <hmelman@gmail.com> said:

    Howard> Thanks for all this info.  So on that page, in the second headed section of the 
    Howard> table "Emoji Font" is where U+1F37D appears.  In the "text-vs" row, which
    Howard> I think is the case of a lone U+1F37D, I see the emoji glyph in my mac browser.
    Howard> The description in that header says:

    Howard>   “text+ts” should be monochrome; everything else should be colorful & monospace.

    Howard> which matches what I see.  So I think, a lone U+1F37D should be displayed
    Howard> as an "emoji glyph".  

_If_ you've specified an emoji font for it, which we donʼt do by
default, since it has Emoji_Presentation = False, so you should look
at the "Plain" section instead.

    Howard> Can emacs be configured to display these lone codepoints via my emoji font?
    Howard> I gather that's what using the 'symbol script does but also includes more.
    Howard> Can I (or emacs out-of-the-box) be more selective in the call to 
    Howard> set-fontset-font or some other api?

Yes. Try:

(set-fontset-font t #x1f37d
    '("Apple Color Emoji" . "iso10646-1") nil 'prepend)

For a range of codepoints, replace #x1f37d with something like
'(#x1f37d . #x1f3aa)

    >>>> I said, I don't understand this stuff.  Is this extra codepoint supposed
    >>>> to be added for me?  It doesn't seem like other apps require it.
    >> 
    Eli> Emacs currently doesn't insert the variation selectors automatically,
    Eli> although perhaps the Emoji input method should (or maybe already
    Eli> does, I didn't check).
    >> 
    >> The stuff Lars added on master puts in variation selectors where
    >> needed.

    Howard> The emoji input method isn't on 28 so I can't check, but FWIW this seems
    Howard> to not match the behavior I see using the mac system emoji picker
    Howard> which seems to just insert a lone U+1F37D when I pick this emoji.

I donʼt think we should follow what the mac does when it contradicts
what Unicode is telling us.

    Howard> And I'll add, if that's displayed equivalently I'd prefer it, because I wouldn't
    Howard> have to deal with "extra invisible characters" after the glyph when
    Howard> using emacs editing commands (unless this is different behavior in 29
    Howard> than in 28 when I add the variation selector character).

Those characters get composed, so they get treated as a single
unit. They really donʼt cause any problems.

    >> Modulo `use-default-font-for-symbols'

    Howard> FWIW this variable set to t for me which I think is the default.

I meant you should try setting it to 'nil'.

    Eli> So I think the recipe in NEWS is correct, and your expectations were
    Eli> inconsistent with the Emacs support for Emoji, at least with its state
    Eli> in Emacs 28.1.
    >> 
    >> Iʼm not sure what we could change. I guess we could add a
    >> configuration variable that says 'treat every code point that has a
    >> default text presentation and an emoji one as emoji', except we
    >> already have that: VARIATION SELECTOR 16

    Howard> I think the section I mentioned above is this case, that things in the
    Howard> "emojiFont" grouping, w/o a variation selector should be presented
    Howard> colorful and monospace (which I take to mean "as emoji").  Am I
    Howard> misunderstanding?

Again: when an emoji font has been specified for the codepoint.

Robert
-- 

bug#54970: 28.1; Some emoji no longer display

[Eli Zaretskii]

> From: Howard Melman <hmelman@gmail.com>
> Date: Sun, 17 Apr 2022 10:49:35 -0400
> 
> 
> > On Apr 17, 2022, at 10:44 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> > 
> > Emacs also displays tofu if it finds no suitable font.  What you
> > describe sounds like Emacs found some font that claimed to have a
> > glyph for this codepoint, but the glyph displays as blank.  Which is,
> > of course, in contradiction with the "no font available" part, so I
> > admit I don't understand what happened on your system.  Was that in
> > "emacs -Q"?
> 
> Yes it was.

Does this empty display happen with every character for which you have
no fonts?  For example, what about this string: 𓂋𓏤𓈖𓆎𓅓‌𓏏𓊖?

On my system, characters for which Emacs cannot find a font display as
"tofu": small squares that show the codepoint of the character in
hex.  That's what should happen by default in those cases.

bug#54970: 28.1; Some emoji no longer display

[Howard Melman]

[-- Attachment #1: Type: text/plain, Size: 1127 bytes --]



> On Apr 17, 2022, at 11:50 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Howard Melman <hmelman@gmail.com>
>> Date: Sun, 17 Apr 2022 10:49:35 -0400
>> 
>> 
>>> On Apr 17, 2022, at 10:44 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>>> 
>>> Emacs also displays tofu if it finds no suitable font.  What you
>>> describe sounds like Emacs found some font that claimed to have a
>>> glyph for this codepoint, but the glyph displays as blank.  Which is,
>>> of course, in contradiction with the "no font available" part, so I
>>> admit I don't understand what happened on your system.  Was that in
>>> "emacs -Q"?
>> 
>> Yes it was.
> 
> Does this empty display happen with every character for which you have
> no fonts?  For example, what about this string: 𓂋𓏤𓈖𓆎𓅓‌𓏏𓊖?
> 
> On my system, characters for which Emacs cannot find a font display as
> "tofu": small squares that show the codepoint of the character in
> hex.  That's what should happen by default in those cases.

I cut and pasted that string from Apple Mail to Emacs 28 -Q on macOS 11.6.5
and saw this:



Howard

[-- Attachment #2.1: Type: text/html, Size: 2029 bytes --]

[-- Attachment #2.2: Screen Shot 2022-04-17 at 11.52.42 AM.png --]
[-- Type: image/png, Size: 12930 bytes --]

bug#54970: 28.1; Some emoji no longer display

[Eli Zaretskii]

> From: Howard Melman <hmelman@gmail.com>
> Date: Sun, 17 Apr 2022 11:54:39 -0400
> Cc: Robert Pluim <rpluim@gmail.com>,
>  Alan Third <alan@idiocy.org>,
>  Lars Ingebrigtsen <larsi@gnus.org>,
>  54970@debbugs.gnu.org
> 
> > Does this empty display happen with every character for which you have
> > no fonts?  For example, what about this string: 𓂋𓏤𓈖𓆎𓅓‌𓏏𓊖?
> > 
> > On my system, characters for which Emacs cannot find a font display as
> > "tofu": small squares that show the codepoint of the character in
> > hex.  That's what should happen by default in those cases.
> 
> I cut and pasted that string from Apple Mail to Emacs 28 -Q on macOS 11.6.5
> and saw this:

So the example didn't work, because you do have a font for those
characters.  Can you find some characters that don't have fonts?  For
example, do any of the scripts in etc/HELLO show as tofu or "blank"?

(I'm trying to understand why you had those blank displays where you
should have seen tofu; if you are not interested in trying to figure
this out further, I'm fully prepared to stop asking questions.)

bug#54970: 28.1; Some emoji no longer display

[Howard Melman]

[-- Attachment #1: Type: text/plain, Size: 668 bytes --]

On Apr 17, 2022, at 12:17 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> So the example didn't work, because you do have a font for those
> characters.  Can you find some characters that don't have fonts?  For
> example, do any of the scripts in etc/HELLO show as tofu or "blank"?
> 
> (I'm trying to understand why you had those blank displays where you
> should have seen tofu; if you are not interested in trying to figure
> this out further, I'm fully prepared to stop asking questions.)

I'm willing.  FYI, I'm using the prebuilt emacs 28 from https://emacsformacosx.com/
I don't build my own.

Here are two images showing the whole HELLO file.




[-- Attachment #2.1: Type: text/html, Size: 1522 bytes --]

[-- Attachment #2.2: Screen Shot 2022-04-17 at 12.47.15 PM.png --]
[-- Type: image/png, Size: 840505 bytes --]

[-- Attachment #2.3: Screen Shot 2022-04-17 at 12.47.41 PM.png --]
[-- Type: image/png, Size: 258733 bytes --]

bug#54970: 28.1; Some emoji no longer display

[Eli Zaretskii]

> From: Howard Melman <hmelman@gmail.com>
> Date: Sun, 17 Apr 2022 12:48:32 -0400
> Cc: Robert Pluim <rpluim@gmail.com>,
>  Alan Third <alan@idiocy.org>,
>  Lars Ingebrigtsen <larsi@gnus.org>,
>  54970@debbugs.gnu.org
> 
> > So the example didn't work, because you do have a font for those
> > characters.  Can you find some characters that don't have fonts?  For
> > example, do any of the scripts in etc/HELLO show as tofu or "blank"?
> > 
> > (I'm trying to understand why you had those blank displays where you
> > should have seen tofu; if you are not interested in trying to figure
> > this out further, I'm fully prepared to stop asking questions.)
> 
> I'm willing.  FYI, I'm using the prebuilt emacs 28 from https://emacsformacosx.com/
> I don't build my own.
> 
> Here are two images showing the whole HELLO file.

You have too many fonts ;-)  It is strange that with so many fonts the
U+1F37D couldn't be displayed.

Okay, try with codepoints between D800 and DB7F: what does that
produce?

bug#54970: 28.1; Some emoji no longer display

[Howard Melman]

On Apr 17, 2022, at 2:00 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> You have too many fonts ;-)  It is strange that with so many fonts the
> U+1F37D couldn't be displayed.

macOS supports lots of languages out of the box :)

I'm reading a lot about unicode and fontsets and trying to wrap
my head around it.  I can't quite answer this yet, the font to display
U+1F37D is Apple Color Emoji.  I get that emacs might not want
to enable that font by default, but emacs does use it by default to show
some emoji, though I don't see it listed when I do M-x describe-fontset.

> Okay, try with codepoints between D800 and DB7F: what does that produce?

I'm sorry I don't follow what you want me to do, can you be more specific?

Howard

bug#54970: 28.1; Some emoji no longer display

[Lars Ingebrigtsen]

Howard Melman <hmelman@gmail.com> writes:

> I'm reading a lot about unicode and fontsets and trying to wrap
> my head around it.  I can't quite answer this yet, the font to display
> U+1F37D is Apple Color Emoji.  I get that emacs might not want
> to enable that font by default, but emacs does use it by default to show
> some emoji, though I don't see it listed when I do M-x describe-fontset.

I haven't been following this thread closely, but has somebody mentioned
that some characters have both a symbol and an emoji expression?  My
guess is that Apple, by default, displays characters like 🍽
"fork and knife with plate" using the emoji expression.  So it has a
font with a definition with the non-emoji expression which is just
blank.

Emacs, by default, uses the symbol expression for the character, unless
you tell Emacs to do something else.

Other characters have only an emoji expression, and then there's no
confusion.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#54970: 28.1; Some emoji no longer display

[Eli Zaretskii]

> From: Howard Melman <hmelman@gmail.com>
> Date: Sun, 17 Apr 2022 14:09:25 -0400
> Cc: Robert Pluim <rpluim@gmail.com>,
>  Alan Third <alan@idiocy.org>,
>  Lars Ingebrigtsen <larsi@gnus.org>,
>  54970@debbugs.gnu.org
> 
> I'm reading a lot about unicode and fontsets and trying to wrap
> my head around it.  I can't quite answer this yet, the font to display
> U+1F37D is Apple Color Emoji.  I get that emacs might not want
> to enable that font by default, but emacs does use it by default to show
> some emoji, though I don't see it listed when I do M-x describe-fontset.

The question is why didn't Emacs find that font when asked to display
U+1F37D, and why did that produce the "blank" display instead of the
expected tofu.

> > Okay, try with codepoints between D800 and DB7F: what does that produce?
> 
> I'm sorry I don't follow what you want me to do, can you be more specific?

"C-x 8 RET d800 RET" and tell what you get.

bug#54970: 28.1; Some emoji no longer display

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Robert Pluim <rpluim@gmail.com>,  Alan
>  Third <alan@idiocy.org>,  54970@debbugs.gnu.org
> Date: Sun, 17 Apr 2022 20:27:41 +0200
> 
> I haven't been following this thread closely, but has somebody mentioned
> that some characters have both a symbol and an emoji expression?  My
> guess is that Apple, by default, displays characters like 🍽
> "fork and knife with plate" using the emoji expression.  So it has a
> font with a definition with the non-emoji expression which is just
> blank.

That's what I thought, but "C-u C-x =" says "no font was found", which
AFAIU contradicts this theory.

bug#54970: 28.1; Some emoji no longer display

[Howard Melman]

[-- Attachment #1: Type: text/plain, Size: 4670 bytes --]



> On Apr 17, 2022, at 11:14 AM, Robert Pluim <rpluim@gmail.com> wrote:
> 
>>>>>> On Sun, 17 Apr 2022 10:35:17 -0400, Howard Melman <hmelman@gmail.com> said:
> 
>    Howard> Thanks for all this info.  So on that page, in the second headed section of the 
>    Howard> table "Emoji Font" is where U+1F37D appears.  In the "text-vs" row, which
>    Howard> I think is the case of a lone U+1F37D, I see the emoji glyph in my mac browser.
>    Howard> The description in that header says:
> 
>    Howard>   “text+ts” should be monochrome; everything else should be colorful & monospace.
> 
>    Howard> which matches what I see.  So I think, a lone U+1F37D should be displayed
>    Howard> as an "emoji glyph".  
> 
> _If_ you've specified an emoji font for it, which we donʼt do by
> default, since it has Emoji_Presentation = False,

Ok, so you find out that U+1F37D has Emoji_Presentation = False from
http://www.unicode.org/Public/emoji/1.0//emoji-data.txt
which indicates the default presentation should be text:

1F37D ;	text ;	L2 ;	none ;	w	# V7.0 (🍽) FORK AND KNIFE WITH PLATE

> so you should look at the "Plain" section instead.

Ok, looking in the plain section under text-vs on macos 11.6.5 in Safari
I see an emoji glyph there too for U+1F37D.  This is the plain section 
and U+1F37D is at the end of the top row in this image. There are a lot
of emoji shown in that section but less than in the emojiFont section.



> 
>    Howard> Can emacs be configured to display these lone codepoints via my emoji font?
>    Howard> I gather that's what using the 'symbol script does but also includes more.
>    Howard> Can I (or emacs out-of-the-box) be more selective in the call to 
>    Howard> set-fontset-font or some other api?
> 
> Yes. Try:
> 
> (set-fontset-font t #x1f37d
>    '("Apple Color Emoji" . "iso10646-1") nil 'prepend)
> 
> For a range of codepoints, replace #x1f37d with something like
> '(#x1f37d . #x1f3aa)

Thanks, doing these definitely gets me further to where I'd like:
(set-fontset-font t '(#x1F170 . #x1F6F3) '("Apple Color Emoji" . "iso10646-1") nil 'prepend)
(set-fontset-font t '(#x2139 . #x3299) '("Apple Color Emoji" . "iso10646-1") nil 'prepend)

I'm still confused as to why the above works but this didn't:

(set-fontset-font t 'emoji '("Apple Color Emoji" . "iso10646-1") nil 'prepend)

And I as I look at script-representative-chars, emoji is defined to be  (emoji 127744 128512)
which I think means the hex range x1F300 - x1F600 so shouldn't include x1f37d?
Or does it not because the default expression is text?  And if so how is that
factored into the emoji script symbol passed to set-fontset-font, I don't see
how that's defined other than as this range. And when I specify a range 
directly I get my pretty glyph displayed.

> I donʼt think we should follow what the mac does when it contradicts
> what Unicode is telling us.

I certainly agree with this.   I see that 
https://unicode.org/reports/tr51/#Emoji_Implementation_Notes
says:

	• only fully-qualified emoji zwj sequences should be generated by keyboards and other user input devices.

and working through the definition of fully-qualified emoji
https://unicode.org/reports/tr51/#def_fully_qualified_emoji
a lone U+1F37D is not fully-qualified.  

If I understand emacs' state correctly, insert-char is doing the right thing
because it's just inserting a character.  I think I'm picking an emoji but
I'm not really, I'm picking a single character (in this case U+1F37D). 
A later Emacs will have an emoji input method that would
be like a real emoji picker that lets me insert a proper fully-qualified sequence.

>    Howard> And I'll add, if that's displayed equivalently I'd prefer it, because I wouldn't
>    Howard> have to deal with "extra invisible characters" after the glyph when
>    Howard> using emacs editing commands (unless this is different behavior in 29
>    Howard> than in 28 when I add the variation selector character).
> 
> Those characters get composed, so they get treated as a single
> unit. They really donʼt cause any problems.

Well C-f and C-b seem to move point between them which is somewhat startling.

>>> Modulo `use-default-font-for-symbols'
> 
>    Howard> FWIW this variable set to t for me which I think is the default.
> 
> I meant you should try setting it to 'nil'.

In an emacs -Q in the scratch buffer I inserted a lone U+1F37D 
Toggling use-default-font-for-symbols had no effect on its display.  
Even after I did:

(set-fontset-font t 'emoji '("Apple Color Emoji" . "iso10646-1") nil 'prepend)

Howard

[-- Attachment #2.1: Type: text/html, Size: 8164 bytes --]

[-- Attachment #2.2: Screen Shot 2022-04-17 at 12.01.23 PM.png --]
[-- Type: image/png, Size: 237089 bytes --]

bug#54970: 28.1; Some emoji no longer display

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> That's what I thought, but "C-u C-x =" says "no font was found", which
> AFAIU contradicts this theory.

Huh, indeed.  How odd.  Why isn't Emacs showing tofu instead of a blank
square, then?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#54970: 28.1; Some emoji no longer display

[Howard Melman]

[-- Attachment #1: Type: text/plain, Size: 1089 bytes --]



> On Apr 17, 2022, at 2:34 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> From: Howard Melman <hmelman@gmail.com>
>> Date: Sun, 17 Apr 2022 14:09:25 -0400
>> Cc: Robert Pluim <rpluim@gmail.com>,
>> Alan Third <alan@idiocy.org>,
>> Lars Ingebrigtsen <larsi@gnus.org>,
>> 54970@debbugs.gnu.org
>> 
>> I'm reading a lot about unicode and fontsets and trying to wrap
>> my head around it.  I can't quite answer this yet, the font to display
>> U+1F37D is Apple Color Emoji.  I get that emacs might not want
>> to enable that font by default, but emacs does use it by default to show
>> some emoji, though I don't see it listed when I do M-x describe-fontset.
> 
> The question is why didn't Emacs find that font when asked to display
> U+1F37D, and why did that produce the "blank" display instead of the
> expected tofu.
> 
>>> Okay, try with codepoints between D800 and DB7F: what does that produce?
>> 
>> I'm sorry I don't follow what you want me to do, can you be more specific?
> 
> "C-x 8 RET d800 RET" and tell what you get.



Sorry, same thing.  db7f is the same as well.

Hoawrd



[-- Attachment #2.1: Type: text/html, Size: 2284 bytes --]

[-- Attachment #2.2: Screen Shot 2022-04-17 at 2.45.29 PM.png --]
[-- Type: image/png, Size: 11158 bytes --]

bug#54970: 28.1; Some emoji no longer display

[Lars Ingebrigtsen]

Howard Melman <hmelman@gmail.com> writes:

> I'm still confused as to why the above works but this didn't:
>
> (set-fontset-font t 'emoji '("Apple Color Emoji" . "iso10646-1") nil 'prepend)

This works for me:

(set-fontset-font t 'symbol '("Apple Color Emoji") nil 'prepend)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#54970: 28.1; Some emoji no longer display

[Howard Melman]

> On Apr 17, 2022, at 2:45 PM, Lars Ingebrigtsen <larsi@gnus.org> wrote:
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
>> That's what I thought, but "C-u C-x =" says "no font was found", which
>> AFAIU contradicts this theory.
> 
> Huh, indeed.  How odd.  Why isn't Emacs showing tofu instead of a blank
> square, then?

FWIW the macport of Emacs 28 displays tofu for me for the book character.
Below is C-u C-x = for it.  The same character displays as blank in vanilla 
Emacs 28.1 -Q  (with the same C-u C-x = output other than the hl-line property) 

==================

             position: 1 of 3 (0%), column: 0
            character: 🕮 (displayed as 🕮) (codepoint 128366, #o372556, #x1f56e)
              charset: unicode (Unicode (ISO10646))
code point in charset: 0x1F56E
               script: symbol
               syntax: w 	which means: word
             category: .:Base
             to input: type "C-x 8 RET 1f56e" or "C-x 8 RET BOOK"
          buffer code: #xF0 #x9F #x95 #xAE
            file code: #xF0 #x9F #x95 #xAE (encoded by coding system utf-8-unix)
              display: no font available

Character code properties: customize what to show
  name: BOOK
  general-category: So (Symbol, Other)
  decomposition: (128366) ('🕮')

There are 2 overlays here:
 From 1 to 3
  face                 hl-line
  priority             -50
  window               #<window 3 on *scratch*>
 From 1 to 1
  display              ""


There are text properties here:
  fontified            nil

bug#54970: 28.1; Some emoji no longer display

[Howard Melman]

> On Apr 17, 2022, at 2:49 PM, Lars Ingebrigtsen <larsi@gnus.org> wrote:
> 
> Howard Melman <hmelman@gmail.com> writes:
> 
>> I'm still confused as to why the above works but this didn't:
>> 
>> (set-fontset-font t 'emoji '("Apple Color Emoji" . "iso10646-1") nil 'prepend)
> 
> This works for me:
> 
> (set-fontset-font t 'symbol '("Apple Color Emoji") nil 'prepend)

Yes, it does for me too.  But this started because the NEWS file said
to use 'emoji and that didn't work.  And as I've learned, I probably
don't want to set all symbols to use Apple Color Emoji.  But I'm still
unclear as to why 'emoji didn't work.

Howard

bug#54970: 28.1; Some emoji no longer display

[Eli Zaretskii]

> From: Howard Melman <hmelman@gmail.com>
> Date: Sun, 17 Apr 2022 14:44:25 -0400
> Cc: Eli Zaretskii <eliz@gnu.org>,
>  Alan Third <alan@idiocy.org>,
>  Lars Ingebrigtsen <larsi@gnus.org>,
>  54970@debbugs.gnu.org
> 
> I'm still confused as to why the above works but this didn't:
> 
> (set-fontset-font t 'emoji '("Apple Color Emoji" . "iso10646-1") nil 'prepend)

Because that character is not in the emoji script:

  (aref char-script-table #x1f37d) => symbol

> And I as I look at script-representative-chars, emoji is defined to be  (emoji 127744 128512)
> which I think means the hex range x1F300 - x1F600 so shouldn't include x1f37d?

You misinterpret script-representative-chars.  That doesn't mean the
range of characters between the values, but only the discrete examples
of the script's characters, which Emacs uses to see if a font supports
a script well enough to use it.  IOW, only those specific codepoints
are used as representatives, not any others.

> Or does it not because the default expression is text?  And if so how is that
> factored into the emoji script symbol passed to set-fontset-font, I don't see
> how that's defined other than as this range.

See lisp/international/charscript.el for how we assign characters to
scripts.  (It's a file generated from the UCD.)

> > Those characters get composed, so they get treated as a single
> > unit. They really donʼt cause any problems.
> 
> Well C-f and C-b seem to move point between them which is somewhat startling.

No, they shouldn't.  If they do, it means you don't have character
composition working.  "C-u C-x =" should describe the composition of
it happened.

> >>> Modulo `use-default-font-for-symbols'
> > 
> >    Howard> FWIW this variable set to t for me which I think is the default.
> > 
> > I meant you should try setting it to 'nil'.
> 
> In an emacs -Q in the scratch buffer I inserted a lone U+1F37D 
> Toggling use-default-font-for-symbols had no effect on its display.  

It will only have effect if the default font has a glyph for that
character.

bug#54970: 28.1; Some emoji no longer display

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: hmelman@gmail.com,  rpluim@gmail.com,  alan@idiocy.org,
>   54970@debbugs.gnu.org
> Date: Sun, 17 Apr 2022 20:45:35 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > That's what I thought, but "C-u C-x =" says "no font was found", which
> > AFAIU contradicts this theory.
> 
> Huh, indeed.  How odd.  Why isn't Emacs showing tofu instead of a blank
> square, then?

No idea.  Maybe it's a macOS thing or something?

bug#54970: 28.1; Some emoji no longer display

[Eli Zaretskii]

> From: Howard Melman <hmelman@gmail.com>
> Date: Sun, 17 Apr 2022 14:46:25 -0400
> Cc: Robert Pluim <rpluim@gmail.com>,
>  Alan Third <alan@idiocy.org>,
>  Lars Ingebrigtsen <larsi@gnus.org>,
>  54970@debbugs.gnu.org
> 
> > "C-x 8 RET d800 RET" and tell what you get.
> 
> 
> 
> Sorry, same thing.  db7f is the same as well.

And if you customize glyphless-char-display-control to something
non-default, does it affect how these codepoints are displayed?

bug#54970: 28.1; Some emoji no longer display

[Alan Third]

On Sun, Apr 17, 2022 at 09:58:56PM +0300, Eli Zaretskii wrote:
> > From: Lars Ingebrigtsen <larsi@gnus.org>
> > Cc: hmelman@gmail.com,  rpluim@gmail.com,  alan@idiocy.org,
> >   54970@debbugs.gnu.org
> > Date: Sun, 17 Apr 2022 20:45:35 +0200
> > 
> > Eli Zaretskii <eliz@gnu.org> writes:
> > 
> > > That's what I thought, but "C-u C-x =" says "no font was found", which
> > > AFAIU contradicts this theory.
> > 
> > Huh, indeed.  How odd.  Why isn't Emacs showing tofu instead of a blank
> > square, then?
> 
> No idea.  Maybe it's a macOS thing or something?

I'm pretty sure that I've never ever seen NS Emacs on macOS produce
anything but a blank square (and it's always a square) for missing
characters.

I don't understand macfont.m, or Emacs's font handling, well enough to
really know what's going on.

Looking at the limited number of macOS apps I have, it looks like they
all display something different, so I guess the output is application
dependent.

-- 
Alan Third

bug#54970: 28.1; Some emoji no longer display

[Howard Melman]

Thanks for the elided info, I get it now.

> On Apr 17, 2022, at 2:58 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> 
>> Well C-f and C-b seem to move point between them which is somewhat startling.
> 
> No, they shouldn't.  If they do, it means you don't have character
> composition working.  "C-u C-x =" should describe the composition of
> it happened.

Yes I see that.  I swear that at the beginning of this when I first used
insert-char to add FE0F after 1f37d they didn't compose, but I
can't reproduce that now.

>> In an emacs -Q in the scratch buffer I inserted a lone U+1F37D 
>> Toggling use-default-font-for-symbols had no effect on its display.  
> 
> It will only have effect if the default font has a glyph for that character.

Ah I was confusing default font with default fontset.

And I gather that something is loading Apple Color Emoji into my
fontset when I type something emacs thinks it can display.
Because if I do describe-fontset in a new emacs -Q Apple Color Emoji
isn't there, but if I insert the man emoji (which displays) and do
describe-fontset again Apple Color Emoji is there.

Thanks, I think I'm caught up now. Thanks for the all the handholding.

Howard

bug#54970: 28.1; Some emoji no longer display

[Alan Third]

On Sun, Apr 17, 2022 at 08:27:45PM +0100, Alan Third wrote:
> On Sun, Apr 17, 2022 at 09:58:56PM +0300, Eli Zaretskii wrote:
> > No idea.  Maybe it's a macOS thing or something?
> 
> I'm pretty sure that I've never ever seen NS Emacs on macOS produce
> anything but a blank square (and it's always a square) for missing
> characters.

Hmm, in fact, is this, from nsterm.m, the culprit?

    case GLYPHLESS_GLYPH:
      if (s->for_overlaps || (s->cmp_from > 0
			      && ! s->first_glyph->u.cmp.automatic))
        s->background_filled_p = 1;
      else
        ns_maybe_dumpglyphs_background
          (s, s->first_glyph->type == COMPOSITE_GLYPH);
      /* ... */
      /* Not yet implemented.  */
      /* ... */
      break;

-- 
Alan Third

bug#54970: 28.1; Some emoji no longer display

[Howard Melman]

On Apr 17, 2022, at 3:00 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
> And if you customize glyphless-char-display-control to something
> non-default, does it affect how these codepoints are displayed?

It does not seem to.  I put the BOOK character in *scratch* and
opened customize-option for glyphless-char-display-control
and changed the no-font option several times using 
Set For Current Session and didn't see the character in
*scratch* change at all.

Howard

bug#54970: 28.1; Some emoji no longer display

[Alan Third]

[-- Attachment #1: Type: text/plain, Size: 1150 bytes --]

On Sun, Apr 17, 2022 at 08:47:07PM +0100, Alan Third wrote:
> On Sun, Apr 17, 2022 at 08:27:45PM +0100, Alan Third wrote:
> > On Sun, Apr 17, 2022 at 09:58:56PM +0300, Eli Zaretskii wrote:
> > > No idea.  Maybe it's a macOS thing or something?
> > 
> > I'm pretty sure that I've never ever seen NS Emacs on macOS produce
> > anything but a blank square (and it's always a square) for missing
> > characters.
> 
> Hmm, in fact, is this, from nsterm.m, the culprit?
> 
>     case GLYPHLESS_GLYPH:
>       if (s->for_overlaps || (s->cmp_from > 0
> 			      && ! s->first_glyph->u.cmp.automatic))
>         s->background_filled_p = 1;
>       else
>         ns_maybe_dumpglyphs_background
>           (s, s->first_glyph->type == COMPOSITE_GLYPH);
>       /* ... */
>       /* Not yet implemented.  */
>       /* ... */
>       break;

To answer my own question: yes. Yes it is.

Patch attached.

It doesn't look to me like it displays quite right, the right hand
side of the hex code gets cut off on my Mac and under GNUstep only the
first line is displayed, but I can't see any reason for it.

Maybe this is better than nothing, though.

-- 
Alan Third

[-- Attachment #2: 0001-Fix-glyphless-glyph-display-on-NS-bug-54970.patch --]
[-- Type: text/x-diff, Size: 3570 bytes --]

From a3f75f63fed67e4a63eb1412d910c9e66c4cc2b8 Mon Sep 17 00:00:00 2001
From: Alan Third <alan@idiocy.org>
Date: Sun, 17 Apr 2022 21:15:05 +0100
Subject: [PATCH] Fix glyphless glyph display on NS (bug#54970)

* src/nsterm.m (ns_draw_glyphless_glyph_string_foreground): New
function.
(ns_draw_glyph_string): Use the new function.
---
 src/nsterm.m | 79 ++++++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 76 insertions(+), 3 deletions(-)

diff --git a/src/nsterm.m b/src/nsterm.m
index 550f29212e..18e219949b 100644
--- a/src/nsterm.m
+++ b/src/nsterm.m
@@ -3945,6 +3945,81 @@ Function modeled after x_draw_glyph_string_box ().
     }
 }
 
+/* Draw the foreground of glyph string S for glyphless characters.  */
+static void
+ns_draw_glyphless_glyph_string_foreground (struct glyph_string *s)
+{
+  struct glyph *glyph = s->first_glyph;
+  unsigned char2b[8];
+  int x, i, j;
+
+  /* If first glyph of S has a left box line, start drawing the text
+     of S to the right of that box line.  */
+  if (s->face && s->face->box != FACE_NO_BOX
+      && s->first_glyph->left_box_line_p)
+    x = s->x + max (s->face->box_vertical_line_width, 0);
+  else
+    x = s->x;
+
+  s->char2b = char2b;
+
+  for (i = 0; i < s->nchars; i++, glyph++)
+    {
+      char buf[7];
+      char *str = NULL;
+      int len = glyph->u.glyphless.len;
+
+      if (glyph->u.glyphless.method == GLYPHLESS_DISPLAY_ACRONYM)
+	{
+	  if (len > 0
+	      && CHAR_TABLE_P (Vglyphless_char_display)
+	      && (CHAR_TABLE_EXTRA_SLOTS (XCHAR_TABLE (Vglyphless_char_display))
+		  >= 1))
+	    {
+	      Lisp_Object acronym
+		= (! glyph->u.glyphless.for_no_font
+		   ? CHAR_TABLE_REF (Vglyphless_char_display,
+				     glyph->u.glyphless.ch)
+		   : XCHAR_TABLE (Vglyphless_char_display)->extras[0]);
+	      if (STRINGP (acronym))
+		str = SSDATA (acronym);
+	    }
+	}
+      else if (glyph->u.glyphless.method == GLYPHLESS_DISPLAY_HEX_CODE)
+	{
+	  unsigned int ch = glyph->u.glyphless.ch;
+	  eassume (ch <= MAX_CHAR);
+	  sprintf (buf, "%0*X", ch < 0x10000 ? 4 : 6, ch);
+	  str = buf;
+	}
+
+      if (str)
+	{
+	  int upper_len = (len + 1) / 2;
+
+	  /* It is assured that all LEN characters in STR is ASCII.  */
+	  for (j = 0; j < len; j++)
+            char2b[j] = s->font->driver->encode_char (s->font, str[j]) & 0xFFFF;
+	  s->font->driver->draw (s, 0, upper_len,
+				 x + glyph->slice.glyphless.upper_xoff,
+				 s->ybase + glyph->slice.glyphless.upper_yoff,
+				 false);
+	  s->font->driver->draw (s, upper_len, len,
+				 x + glyph->slice.glyphless.lower_xoff,
+				 s->ybase + glyph->slice.glyphless.lower_yoff,
+				 false);
+	}
+      if (glyph->u.glyphless.method != GLYPHLESS_DISPLAY_THIN_SPACE)
+        ns_draw_box (NSMakeRect (x, s->ybase - glyph->ascent,
+                                 glyph->pixel_width - 1,
+                                 glyph->ascent + glyph->descent - 1),
+                     1, 1,
+                     [NSColor colorWithUnsignedLong:NS_FACE_FOREGROUND (s->face)],
+                     YES, YES);
+      x += glyph->pixel_width;
+   }
+}
+
 static void
 ns_draw_glyph_string (struct glyph_string *s)
 /* --------------------------------------------------------------------------
@@ -4058,9 +4133,7 @@ Function modeled after x_draw_glyph_string_box ().
       else
         ns_maybe_dumpglyphs_background
           (s, s->first_glyph->type == COMPOSITE_GLYPH);
-      /* ... */
-      /* Not yet implemented.  */
-      /* ... */
+      ns_draw_glyphless_glyph_string_foreground (s);
       break;
 
     default:
-- 
2.35.1

bug#54970: 28.1; Some emoji no longer display

[Howard Melman]

> On Apr 17, 2022, at 4:34 PM, Alan Third <alan@idiocy.org> wrote:
> 
> Patch attached.

Glad there's progress, but I don't build my own emacs so
I can't apply this patch to test it.

Howard

bug#54970: 28.1; Some emoji no longer display

[Howard Melman]

So back to this:

> On Apr 17, 2022, at 9:34 AM, Robert Pluim <rpluim@gmail.com> wrote:
> 
>    Eli> No, that wouldn't be right.  We introduced the special 'emoji'
>    Eli> pseudo-script in Emacs 28 to solve the problems of being unable to
>    Eli> distinguish between Emoji codepoints and the other symbols and
>    Eli> punctuation.  We definitely do NOT want to use an Emoji font for
>    Eli> symbols and punctuation characters and character sequences that aren't
>    Eli> Emoji.  Moreover, configuring the fontset to use some font for the
>    Eli> 'symbol' pseudo-script by default doesn't do what you expect, because
>    Eli> Emacs uses the default font for symbols for which the default font has
>    Eli> a glyph.
> 
> Modulo `use-default-font-for-symbols'
> 
>    Eli> So I think the recipe in NEWS is correct, and your expectations were
>    Eli> inconsistent with the Emacs support for Emoji, at least with its state
>    Eli> in Emacs 28.1.
> 
> Iʼm not sure what we could change. I guess we could add a
> configuration variable that says 'treat every code point that has a
> default text presentation and an emoji one as emoji', except we
> already have that: VARIATION SELECTOR 16

Yes, I think I would like a configuration variable like that.  It's seems like a good
application of the Robustness Principle.  

I also thought perhaps there should be another "script" parallel to 'emoji that
included these so I could something like

(set-fontset-font t 'emoji-vs '("Apple Color Emoji" . "iso10646-1") nil 'prepend)

but now I suspect that's not really the right way? Or is it?

Howard

bug#54970: 28.1; Some emoji no longer display

[Eli Zaretskii]

> Date: Sun, 17 Apr 2022 21:34:52 +0100
> From: Alan Third <alan@idiocy.org>
> 
> >     case GLYPHLESS_GLYPH:
> >       if (s->for_overlaps || (s->cmp_from > 0
> > 			      && ! s->first_glyph->u.cmp.automatic))
> >         s->background_filled_p = 1;
> >       else
> >         ns_maybe_dumpglyphs_background
> >           (s, s->first_glyph->type == COMPOSITE_GLYPH);
> >       /* ... */
> >       /* Not yet implemented.  */
> >       /* ... */
> >       break;
> 
> To answer my own question: yes. Yes it is.
> 
> Patch attached.
> 
> It doesn't look to me like it displays quite right, the right hand
> side of the hex code gets cut off on my Mac and under GNUstep only the
> first line is displayed, but I can't see any reason for it.

Can you show screenshots?  I'm not sure I understand the description
of the display problems that this code yields.

The font used for the hex code should be a smaller one, and that is
determined elsewhere in the display code, so maybe that part also
needs some fix in the NS build?

Thanks.

bug#54970: 28.1; Some emoji no longer display

[Eli Zaretskii]

> From: Howard Melman <hmelman@gmail.com>
> Date: Sun, 17 Apr 2022 18:03:38 -0400
> Cc: Eli Zaretskii <eliz@gnu.org>,
>  alan@idiocy.org,
>  larsi@gnus.org,
>  54970@debbugs.gnu.org
> 
> > Iʼm not sure what we could change. I guess we could add a
> > configuration variable that says 'treat every code point that has a
> > default text presentation and an emoji one as emoji', except we
> > already have that: VARIATION SELECTOR 16
> 
> Yes, I think I would like a configuration variable like that.  It's seems like a good
> application of the Robustness Principle.  

We decided not to use the Emoji presentation of these characters by
default for a good reason: there are many symbols that _can_ have
Emoji presentation, but they are used frequently in "normal" text.
Look at the beginning of admin/unidata/emoji-data.txt, and you will
see what I mean: even ASCII characters like '#' and digits can have
Emoji presentation, and we definitely don't want them appear as Emoji.
Don't forget that Emacs display features like this one are _global_,
so if for some insane reason we decide to have digits displayed as
Emoji, you will get that everywhere, including on the mode line, for
example.

So what will that hypothetical variable do?  It cannot affect all the
characters that _may_ have the Emoji presentation, so we will need to
decide which ones to affect and which ones not to affect.  If you look
at emoji-data.txt, you will realize that the decision is not easy; for
example, what about U+2122 TRADE MARK SIGN or U+23F0 ALARM CLOCK or
U+262E PEACE SYMBOL?  Some people will want them as Emoji, while
others won't.  So we'd need another variable with a long list of
codepoints that are Emoji by default?  Such a variable sounds like a
PITA to let users customize.

So we decided not to do any of that, and instead to go with the
default presentation as determined by Unicode.  I hope you now
understand better why we did that.

> I also thought perhaps there should be another "script" parallel to 'emoji that
> included these so I could something like
> 
> (set-fontset-font t 'emoji-vs '("Apple Color Emoji" . "iso10646-1") nil 'prepend)
> 
> but now I suspect that's not really the right way? Or is it?

A character cannot belong to more than one script in Emacs, so that's
not possible, AFAIU.  And I don't see why it would be necessary, since
one can customize the fontset for individual codepoints without using
a script symbol.

bug#54970: 28.1; Some emoji no longer display

[Alan Third]

[-- Attachment #1: Type: text/plain, Size: 1678 bytes --]

On Mon, Apr 18, 2022 at 07:29:00AM +0300, Eli Zaretskii wrote:
> > Date: Sun, 17 Apr 2022 21:34:52 +0100
> > From: Alan Third <alan@idiocy.org>
> > 
> > >     case GLYPHLESS_GLYPH:
> > >       if (s->for_overlaps || (s->cmp_from > 0
> > > 			      && ! s->first_glyph->u.cmp.automatic))
> > >         s->background_filled_p = 1;
> > >       else
> > >         ns_maybe_dumpglyphs_background
> > >           (s, s->first_glyph->type == COMPOSITE_GLYPH);
> > >       /* ... */
> > >       /* Not yet implemented.  */
> > >       /* ... */
> > >       break;
> > 
> > To answer my own question: yes. Yes it is.
> > 
> > Patch attached.
> > 
> > It doesn't look to me like it displays quite right, the right hand
> > side of the hex code gets cut off on my Mac and under GNUstep only the
> > first line is displayed, but I can't see any reason for it.
> 
> Can you show screenshots?  I'm not sure I understand the description
> of the display problems that this code yields.
> 
> The font used for the hex code should be a smaller one, and that is
> determined elsewhere in the display code, so maybe that part also
> needs some fix in the NS build?

I've changed my mind about on the Mac. The hex code isn't centred, but
it is readable.

I found a couple of bugs in nsfont.m, which is the GNUstep font
backend, and now the only problem is that the upper and lower text are
drawn in almost the same location. I suspect some error in how the
font backend is calculating the ascent and descent or something. I
don't see what's wrong, though.

Screenshot for GNUstep attached. In this example upper_yoff and
lower_yoff are -14 and -13, respectively.

Po Lu, any ideas?

-- 
Alan Third

[-- Attachment #2: 10FFFF.png --]
[-- Type: image/png, Size: 1167 bytes --]

[-- Attachment #3: v2-0001-Fix-glyphless-glyph-display-on-NS-bug-54970.patch --]
[-- Type: text/x-diff, Size: 4598 bytes --]

From e2c482042bf45a3f7b924b0f298876d84576df1d Mon Sep 17 00:00:00 2001
From: Alan Third <alan@idiocy.org>
Date: Sun, 17 Apr 2022 21:15:05 +0100
Subject: [PATCH v2] Fix glyphless glyph display on NS (bug#54970)

* src/nsterm.m (ns_draw_glyphless_glyph_string_foreground): New
function.
(ns_draw_glyph_string): Use the new function.
* src/nsfont.m (nsfont_draw): Fix the location the glyphs are drawn,
and also which glyphs are drawn.
---
 src/nsfont.m | 10 +++----
 src/nsterm.m | 79 ++++++++++++++++++++++++++++++++++++++++++++++++++--
 2 files changed, 80 insertions(+), 9 deletions(-)

diff --git a/src/nsfont.m b/src/nsfont.m
index f3c8a82930..6915fd217d 100644
--- a/src/nsfont.m
+++ b/src/nsfont.m
@@ -1176,15 +1176,15 @@ is false when (FROM > 0 || TO < S->nchars).  */
 
   face = s->face;
 
-  r.origin.x = s->x;
+  r.origin.x = x;
   if (s->face->box != FACE_NO_BOX && s->first_glyph->left_box_line_p)
     r.origin.x += max (s->face->box_vertical_line_width, 0);
 
-  r.origin.y = s->y;
+  r.origin.y = y;
   r.size.height = FONT_HEIGHT (font);
 
-  for (int i = from; i < to; ++i)
-    c[i] = s->char2b[i];
+  for (int i = 0; i < len; ++i)
+    c[i] = s->char2b[i + from];
 
   /* Fill background if requested.  */
   if (with_background && !isComposite)
@@ -1210,8 +1210,6 @@ is false when (FROM > 0 || TO < S->nchars).  */
     }
 
   /* set up for character rendering */
-  r.origin.y = y;
-
   if (s->hl == DRAW_CURSOR)
     col = FRAME_BACKGROUND_COLOR (s->f);
   else
diff --git a/src/nsterm.m b/src/nsterm.m
index 550f29212e..5a6a4d663b 100644
--- a/src/nsterm.m
+++ b/src/nsterm.m
@@ -3945,6 +3945,81 @@ Function modeled after x_draw_glyph_string_box ().
     }
 }
 
+/* Draw the foreground of glyph string S for glyphless characters.  */
+static void
+ns_draw_glyphless_glyph_string_foreground (struct glyph_string *s)
+{
+  struct glyph *glyph = s->first_glyph;
+  NSGlyph char2b[8];
+  int x, i, j;
+
+  /* If first glyph of S has a left box line, start drawing the text
+     of S to the right of that box line.  */
+  if (s->face && s->face->box != FACE_NO_BOX
+      && s->first_glyph->left_box_line_p)
+    x = s->x + max (s->face->box_vertical_line_width, 0);
+  else
+    x = s->x;
+
+  s->char2b = char2b;
+
+  for (i = 0; i < s->nchars; i++, glyph++)
+    {
+      char buf[7];
+      char *str = NULL;
+      int len = glyph->u.glyphless.len;
+
+      if (glyph->u.glyphless.method == GLYPHLESS_DISPLAY_ACRONYM)
+	{
+	  if (len > 0
+	      && CHAR_TABLE_P (Vglyphless_char_display)
+	      && (CHAR_TABLE_EXTRA_SLOTS (XCHAR_TABLE (Vglyphless_char_display))
+		  >= 1))
+	    {
+	      Lisp_Object acronym
+		= (! glyph->u.glyphless.for_no_font
+		   ? CHAR_TABLE_REF (Vglyphless_char_display,
+				     glyph->u.glyphless.ch)
+		   : XCHAR_TABLE (Vglyphless_char_display)->extras[0]);
+	      if (STRINGP (acronym))
+		str = SSDATA (acronym);
+	    }
+	}
+      else if (glyph->u.glyphless.method == GLYPHLESS_DISPLAY_HEX_CODE)
+	{
+	  unsigned int ch = glyph->u.glyphless.ch;
+	  eassume (ch <= MAX_CHAR);
+	  sprintf (buf, "%0*X", ch < 0x10000 ? 4 : 6, ch);
+	  str = buf;
+	}
+
+      if (str)
+	{
+	  int upper_len = (len + 1) / 2;
+
+	  /* It is assured that all LEN characters in STR is ASCII.  */
+	  for (j = 0; j < len; j++)
+            char2b[j] = s->font->driver->encode_char (s->font, str[j]) & 0xFFFF;
+	  s->font->driver->draw (s, 0, upper_len,
+				 x + glyph->slice.glyphless.upper_xoff,
+				 s->ybase + glyph->slice.glyphless.upper_yoff,
+				 false);
+	  s->font->driver->draw (s, upper_len, len,
+				 x + glyph->slice.glyphless.lower_xoff,
+				 s->ybase + glyph->slice.glyphless.lower_yoff,
+				 false);
+	}
+      if (glyph->u.glyphless.method != GLYPHLESS_DISPLAY_THIN_SPACE)
+        ns_draw_box (NSMakeRect (x, s->ybase - glyph->ascent,
+                                 glyph->pixel_width - 1,
+                                 glyph->ascent + glyph->descent - 1),
+                     1, 1,
+                     [NSColor colorWithUnsignedLong:NS_FACE_FOREGROUND (s->face)],
+                     YES, YES);
+      x += glyph->pixel_width;
+   }
+}
+
 static void
 ns_draw_glyph_string (struct glyph_string *s)
 /* --------------------------------------------------------------------------
@@ -4058,9 +4133,7 @@ Function modeled after x_draw_glyph_string_box ().
       else
         ns_maybe_dumpglyphs_background
           (s, s->first_glyph->type == COMPOSITE_GLYPH);
-      /* ... */
-      /* Not yet implemented.  */
-      /* ... */
+      ns_draw_glyphless_glyph_string_foreground (s);
       break;
 
     default:
-- 
2.35.1

bug#54970: 28.1; Some emoji no longer display

[Howard Melman]

Sigh, I'm sorry the screenshots came out so small, 
my mail program reduced them for space.  If you 
want larger versions of some or all of them I can supply.

Howard

bug#54970: 28.1; Some emoji no longer display

[Eli Zaretskii]

> Date: Mon, 18 Apr 2022 14:19:24 +0100
> From: Alan Third <alan@idiocy.org>
> Cc: larsi@gnus.org, hmelman@gmail.com, rpluim@gmail.com,
> 	54970@debbugs.gnu.org
> 
> I've changed my mind about on the Mac. The hex code isn't centred, but
> it is readable.
> 
> I found a couple of bugs in nsfont.m, which is the GNUstep font
> backend, and now the only problem is that the upper and lower text are
> drawn in almost the same location. I suspect some error in how the
> font backend is calculating the ascent and descent or something. I
> don't see what's wrong, though.
> 
> Screenshot for GNUstep attached. In this example upper_yoff and
> lower_yoff are -14 and -13, respectively.

If the y offsets are wrong, you should step through
produce_glyphless_glyph, where they are calculated.

bug#54970: 28.1; Some emoji no longer display

[Alan Third]

On Mon, Apr 18, 2022 at 05:23:49PM +0300, Eli Zaretskii wrote:
> > Date: Mon, 18 Apr 2022 14:19:24 +0100
> > From: Alan Third <alan@idiocy.org>
> > Cc: larsi@gnus.org, hmelman@gmail.com, rpluim@gmail.com,
> > 	54970@debbugs.gnu.org
> > 
> > I've changed my mind about on the Mac. The hex code isn't centred, but
> > it is readable.
> > 
> > I found a couple of bugs in nsfont.m, which is the GNUstep font
> > backend, and now the only problem is that the upper and lower text are
> > drawn in almost the same location. I suspect some error in how the
> > font backend is calculating the ascent and descent or something. I
> > don't see what's wrong, though.
> > 
> > Screenshot for GNUstep attached. In this example upper_yoff and
> > lower_yoff are -14 and -13, respectively.
> 
> If the y offsets are wrong, you should step through
> produce_glyphless_glyph, where they are calculated.

It turns out that ascent and descent were reversed.

This is one of the biggest problems with the GNUstep port, the ns font
backend was broken even when it was merged in and it's only got
further out of step as time has passed. No wonder the decision was
made, however long ago, to get rid of it on macOS.

Pushed to master.
-- 
Alan Third

bug#54970: 28.1; Some emoji no longer display

[Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Alan Third <alan@idiocy.org> writes:

> It turns out that ascent and descent were reversed.

I thought I fixed all of those last year, but clearly I was wrong.
Thanks for catching that bug.

> This is one of the biggest problems with the GNUstep port, the ns font
> backend was broken even when it was merged in and it's only got
> further out of step as time has passed.

Well, I tried my best.

bug#54970: 28.1; Some emoji no longer display

[Alan Third]

On Tue, Apr 19, 2022 at 09:04:51AM +0800, Po Lu wrote:
> Alan Third <alan@idiocy.org> writes:
> 
> > It turns out that ascent and descent were reversed.
> 
> I thought I fixed all of those last year, but clearly I was wrong.
> Thanks for catching that bug.

I also fixed two other bugs I found. I think there are just so many
small mistakes right through it that we'll never find them all.

> > This is one of the biggest problems with the GNUstep port, the ns font
> > backend was broken even when it was merged in and it's only got
> > further out of step as time has passed.
> 
> Well, I tried my best.

Definitely not your fault! It's much better than it was just a year
ago.
-- 
Alan Third

bug#54970: 28.1; Some emoji no longer display

[Howard Melman]

[-- Attachment #1: Type: text/plain, Size: 1930 bytes --]

I have two follow up questions.

First, is composition supposed to work everywhere?  I have the following
code to add to the C-x 8 map and I use which-key from gnu elpa but its
displayed help isn't composing.  It's just a buffer named " *which-key*"
and I can switch to it but can't do anything in it to inspect things.  It's hard
to follow how they're writing to it.  I can open a which-key bug but I wanted
to understand more about composing and if it's supposed to be enabled
everywhere (which-key doesn't reference it at all)



Second, I'm using this now for emoji fonts:

(set-fontset-font t 'symbol '("Apple Color Emoji" . "iso10646-1") nil 'prepend)

I saw from a related reddit post that the font Symbola has some other glyphs
that seemed of interest.  It renders things as plain graphics not as color
emoji which is fine (or preferred) for math symbols.  I'd like to use Apple
Color Emoji first and then if it doesn't support a character fall back on
Symbola.  I thought to do that with:

(set-fontset-font t 'symbol '("Symbola" . "iso10646-1") nil 'prepend)
(set-fontset-font t 'symbol '("Apple Color Emoji" . "iso10646-1") nil 'prepend)

But it didn't work.  Most emoji were rending as Symbola.  I then removed the
first line and restarted emacs (so just my original line is in the init) and emoji
were *still* rendered with Symbola.  It wasn't until I disabled the font in the
mac utility Font Book.app that it went back to full color emoji rendering. I recall
a similar issue a year ago trying to get Julia Mono working too.

What's the right incantation to do this?  I tried with just the 'emoji script on the 
second call but it didn't help. How and when is emacs searching
for fonts to use and how does it relate to the above set-fontset-font calls? It's
nice that emacs just finds fonts, but it's unnerving that merely installing a font
can break what emacs displays.

Howard

[-- Attachment #2.1: Type: text/html, Size: 3031 bytes --]

[-- Attachment #2.2: Screen Shot 2022-04-22 at 4.33.24 PM.png --]
[-- Type: image/png, Size: 210612 bytes --]

bug#54970: 28.1; Some emoji no longer display

[Eli Zaretskii]

> From: Howard Melman <hmelman@gmail.com>
> Date: Fri, 22 Apr 2022 17:56:34 -0400
> Cc: Robert Pluim <rpluim@gmail.com>,
>  Alan Third <alan@idiocy.org>,
>  Lars Ingebrigtsen <larsi@gnus.org>,
>  54970@debbugs.gnu.org
> 
> First, is composition supposed to work everywhere?

Yes -- as long as the font used for the characters supports that.  In
particular, all the characters of a sequence that's expected to be
composed should be supported by a single font that is used for the
character (usually, but not always, the first character of the
composed sequence) that triggers the composition rule.

> I have the following
> code to add to the C-x 8 map and I use which-key from gnu elpa but its
> displayed help isn't composing.  It's just a buffer named " *which-key*"
> and I can switch to it but can't do anything in it to inspect things.  It's hard
> to follow how they're writing to it.  I can open a which-key bug but I wanted
> to understand more about composing and if it's supposed to be enabled
> everywhere (which-key doesn't reference it at all)

Please discuss this with the developer of which-key, or ask him to
join this discussions and answer your questions about which-key.  That
package is highly non-trivial, weighs in at 2700 lines of Lisp, and
defines several faces (which might affect character composition), so
it isn't easy to understand what it does and how, and isn't really our
job here.  If and when the which-key developer decides there's a core
bug/issue, please come back and show a recipe starting from "emacs -Q"
for reproducing the bug/issue, and we will look into that.

> Second, I'm using this now for emoji fonts:
> 
> (set-fontset-font t 'symbol '("Apple Color Emoji" . "iso10646-1") nil 'prepend)
> 
> I saw from a related reddit post that the font Symbola has some other glyphs
> that seemed of interest.  It renders things as plain graphics not as color
> emoji which is fine (or preferred) for math symbols.  I'd like to use Apple
> Color Emoji first and then if it doesn't support a character fall back on
> Symbola.  I thought to do that with:
> 
> (set-fontset-font t 'symbol '("Symbola" . "iso10646-1") nil 'prepend)
> (set-fontset-font t 'symbol '("Apple Color Emoji" . "iso10646-1") nil 'prepend)
> 
> But it didn't work.  Most emoji were rending as Symbola.  I then removed the
> first line and restarted emacs (so just my original line is in the init) and emoji
> were *still* rendered with Symbola.  It wasn't until I disabled the font in the
> mac utility Font Book.app that it went back to full color emoji rendering. I recall
> a similar issue a year ago trying to get Julia Mono working too.

What was the value of use-default-font-for-symbols?  If it was
non-nil, Emacs will in general try to ignore the fontset when it needs
a font for a character belonging to the 'symbol' pseudo-script.

Also, what does the fontset entry for the relevant character(s) say?
(Invoke "M-x describe-fontset RET fontset-default RET" and look for
the entry that corresponds to the codepoints of the characters you are
interested in.)  Note that the default fontset with which Emacs starts
already includes references to Symbola, even without any
customizations.

And what else is in your init files, besides those two lines?

IOW, you didn't provide enough details to help you understand what
happened in your case.  You didn't even tell what characters you tried
to display, without which it is hard to make any progress here.

> What's the right incantation to do this?

To do what?

> How and when is emacs searching for fonts to use and how does it
> relate to the above set-fontset-font calls?

This question is too general.  In general, Emacs looks for a font when
the default font doesn't have a glyph for a character; it first tries
to find a loaded font that supports the character, and if not found
looks in the fonts specified by the fontsets.

> It's nice that emacs just finds fonts, but it's unnerving that
> merely installing a font can break what emacs displays.

Sorry, I didn't see anything in your report that could be qualified as
"breakage".  Emacs just used a font other than the one you intended to
display some (unnamed) characters, probably because you didn't
sufficiently understand the intricacies of setting up fontsets.  That
doesn't qualify as breakage, IMO.  And yes, setting up fontsets is a
subtle art.

bug#54970: 28.1; Some emoji no longer display

[Eli Zaretskii]

> From: Howard Melman <hmelman@gmail.com>
> Date: Fri, 22 Apr 2022 17:56:34 -0400
> Cc: Robert Pluim <rpluim@gmail.com>,
>  Alan Third <alan@idiocy.org>,
>  Lars Ingebrigtsen <larsi@gnus.org>,
>  54970@debbugs.gnu.org
> 
> First, is composition supposed to work everywhere?  I have the following
> code to add to the C-x 8 map and I use which-key from gnu elpa but its
> displayed help isn't composing.  It's just a buffer named " *which-key*"
> and I can switch to it but can't do anything in it to inspect things.

I'm not sure I understand the difficulty here.  Maybe you didn't set
cursor-type to t in that buffer?

bug#54970: 28.1; Some emoji no longer display

[Howard Melman]

> On Apr 23, 2022, at 3:51 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> First, is composition supposed to work everywhere?
> 
> Yes -- as long as the font used for the characters supports that. 

> Please discuss this with the developer of which-key

Ok, I've opened a bug with which-key to do that.  Thanks.
I had also found the which-key code involved but didn't 
find any references to composition in it or other obvious
things it did to disable it and wanted to confirm that it 
probably shouldn't have to do anything to enable it.

>> Second, I'm using this now for emoji fonts:
>> 
>> (set-fontset-font t 'symbol '("Apple Color Emoji" . "iso10646-1") nil 'prepend)
>> 
>> I saw from a related reddit post that the font Symbola has some other glyphs
>> that seemed of interest.  It renders things as plain graphics not as color
>> emoji which is fine (or preferred) for math symbols.  I'd like to use Apple
>> Color Emoji first and then if it doesn't support a character fall back on
>> Symbola.  I thought to do that with:
>> 
>> (set-fontset-font t 'symbol '("Symbola" . "iso10646-1") nil 'prepend)
>> (set-fontset-font t 'symbol '("Apple Color Emoji" . "iso10646-1") nil 'prepend)
>> 
>> But it didn't work.  Most emoji were rending as Symbola.  I then removed the
>> first line and restarted emacs (so just my original line is in the init) and emoji
>> were *still* rendered with Symbola.  It wasn't until I disabled the font in the
>> mac utility Font Book.app that it went back to full color emoji rendering. I recall
>> a similar issue a year ago trying to get Julia Mono working too.
> 
> What was the value of use-default-font-for-symbols?  If it was
> non-nil, Emacs will in general try to ignore the fontset when it needs
> a font for a character belonging to the 'symbol' pseudo-script.

It is t, which I believe is the default value.  The manuals don't mention
use-default-font-for-symbols so I didn't know about that. 

> Also, what does the fontset entry for the relevant character(s) say?
> (Invoke "M-x describe-fontset RET fontset-default RET" and look for
> the entry that corresponds to the codepoints of the characters you are
> interested in.)  Note that the default fontset with which Emacs starts
> already includes references to Symbola, even without any
> customizations.
> 
> And what else is in your init files, besides those two lines?

I can reproduce this in emacs -Q so the answer is nothing.

Sorry I didn't mention a specific character, but it was
a lot of them and it's hard to lookup codepoints
for each of them.  Here are ones I saw from a file:

🥇 🥈 🥉 💰 ✅ 👍️ 👎️ 👋️ 🤦‍♂️️ 🤷‍♂️ ⭐️ 📺️ 🎥 ♠️ ♣️ ♥️ ♦️
😡 😢 🙄 😂 🤣 😀 🤔 😉
1️⃣ 2️⃣ 3️⃣ 4️⃣ 5️⃣ 6️⃣ 7️⃣ 8️⃣ 9️⃣ 0️⃣
🔨 🎯 🗡 ❄️ 🔥 ⚡️ ☠️ 💥 🙏🏻 👉 🔭️ 💪 ⏳ 😖 ⏩️ 👬 ⏪️ ♻️

And there were may others I saw doing insert-char with
a completion system that showed and rendered the characters.

So I worked with 0x1F528 (HAMMER) a level 1 emoji who's
default rendering is emoji, not text.  Checking describe-fontset
for it I see:

📿 .. 🔽 (#x1F4FF .. #x1F53D)
    -*-Noto Color Emoji-*-*-*-*-*-*-*-*-*-*-iso10646-1
    -*-Symbola-*-*-*-*-*-*-*-*-*-*-iso10646-1
    -*-*-*-*-*-*-*-*-*-*-*-*-iso10646-1

I didn't realize emacs knew about Symbola (why would I?).
And I had forgotten that the scripts symbol and emoji do
not overlap at all (because from the previous issue setting
the emoji script to use Apple Color Emoji (ACE) didn't cover many
characters but setting it for symbol did cover them and seemingly
every emoji.

It turns out HAMMER is in script emoji, so setting the fontset
for 'symbol didn't tell emacs to use ACE for it.  With Symbola
enabled emacs found it and used it.  Without Symbola enabled
emacs didn't find it and I guess searched for a font that includes
this character and found ACE.

So now I'm doing this:

(set-fontset-font t 'symbol '("Apple Color Emoji" . "iso10646-1") nil 'prepend)
(set-fontset-font t 'emoji '("Apple Color Emoji" . "iso10646-1") nil 'prepend)

and it seems to work.  I see emojis I've previously used rendered
in ACE and I see some characters that ACE doesn't cover, like
0x 2b59 (HEAVY CIRCLED SALTIRE) rendered in Symbola.
With Symbola enabled I now need to use both because emacs
can find a font other than ACE that can render script emoji.

I know the definition of the set of emojis is difficult but also that it
doesn't overlap with the intuition of a user not steeped in Unicode. 
Perhaps this is particularly bad on a mac where ACE is used 
universally to render emoji and the system emoji picker (shared
with iOS) includes as emoji many characters in both the emoji
and symbol scripts.  And I understand that you don't want to include
references to ACE in emacs.

I think I would have benefited from a describe-scripts command
that some how showed what ranges these scripts cover in a way
easier than using C-u C-x = for each character.  Or maybe a
function scripts-used-in-region that returned a list of the scripts
of each character.

I do think I and other mac users would benefit from a previously
mentioned possibility of a setting handling those emoji who's default 
rendering is text to either render as emoji or include them in
the emoji script (instead of symbol).

I'm not really clear on categories in emacs.  Maybe an emoji
category that included characters from both script emoji and
some from script symbol and a way to use it to modify a
fontset?  The Categories section of the elisp manual doesn't
say where emacs uses them but I see them used in properties
and regexps and an easy way to regexp search for any character
in an emoji category would be nice.

Back to the original post in this bug, the NEWS instructions to:
    (set-fontset-font t 'emoji '("My New Emoji Font" . "iso10646-1") nil 'prepend)
confused me because it didn't seem to have any effect on macos.
If this (or some other) instruction included "and you may want to do the same
for the script symbol though it might include more characters you
desire" it would have helped.  Particularly if it recommended
this for mac users.  I don't see a mention of emoji in the emacs
manual except in the antinews.  Perhaps it could mention that emacs
by default knows to look for Noto Color Emoji or Symbola for
many emoji characters and how systems without these fonts
might be configured effectively.

Howard

bug#54970: 28.1; Some emoji no longer display

[Eli Zaretskii]

> From: Howard Melman <hmelman@gmail.com>
> Date: Sat, 23 Apr 2022 12:35:47 -0400
> Cc: 54970@debbugs.gnu.org
> 
> I know the definition of the set of emojis is difficult but also that it
> doesn't overlap with the intuition of a user not steeped in Unicode. 
> Perhaps this is particularly bad on a mac where ACE is used 
> universally to render emoji and the system emoji picker (shared
> with iOS) includes as emoji many characters in both the emoji
> and symbol scripts.  And I understand that you don't want to include
> references to ACE in emacs.
> 
> I think I would have benefited from a describe-scripts command
> that some how showed what ranges these scripts cover in a way
> easier than using C-u C-x = for each character.  Or maybe a
> function scripts-used-in-region that returned a list of the scripts
> of each character.

These are all very complex technical issues, and I'm not sure the
right way of letting users customize Emacs in these respects is to
expose all that complexity and advertise it.  We should try finding
easier, more self-explanatory, customization means and options.

> I do think I and other mac users would benefit from a previously
> mentioned possibility of a setting handling those emoji who's default 
> rendering is text to either render as emoji or include them in
> the emoji script (instead of symbol).

Something to consider for the future, I guess.  Emacs 28 is the first
release to include a decent support for Emoji, and we are still
collecting user experience about what we have.

> I'm not really clear on categories in emacs.  Maybe an emoji
> category that included characters from both script emoji and
> some from script symbol and a way to use it to modify a
> fontset?  The Categories section of the elisp manual doesn't
> say where emacs uses them but I see them used in properties
> and regexps and an easy way to regexp search for any character
> in an emoji category would be nice.

Categories are used in regular expressions, but the way fontsets are
implemented, we cannot use categories for controlling font selection,
except very indirectly and unreliably via character composition rules
(which use regexps) -- and that will only affect composed sequences.

> Back to the original post in this bug, the NEWS instructions to:
>     (set-fontset-font t 'emoji '("My New Emoji Font" . "iso10646-1") nil 'prepend)
> confused me because it didn't seem to have any effect on macos.
> If this (or some other) instruction included "and you may want to do the same
> for the script symbol though it might include more characters you
> desire" it would have helped.  Particularly if it recommended
> this for mac users.  I don't see a mention of emoji in the emacs
> manual except in the antinews.  Perhaps it could mention that emacs
> by default knows to look for Noto Color Emoji or Symbola for
> many emoji characters and how systems without these fonts
> might be configured effectively.

NEWS describes what we think is the correct setup for Emoji.  I'm
still not convinced that telling users to customize the font for
'symbol' is a good idea, especially since we prefer symbols and
punctuation to be displayed by the default font as much as possible.
Maybe this contradicts what macOS users expect, but macOS is not the
main platform we want to support.

As for mentioning the fonts for which we set up by default -- there
are others in the default fontset, and I don't see why those two are
to be treated specially.  OTOH, mentioning all of them would be too
much, because there are many, and they differ by platforms.  We should
instead strive to make the default setup work seamlessly for everyone,
IMO.

bug#54970: 28.1; Some emoji no longer display

[Howard Melman]

On Apr 23, 2022, at 12:58 PM, Eli Zaretskii <eliz@gnu.org> wrote:

> These are all very complex technical issues, and I'm not sure the
> right way of letting users customize Emacs in these respects is to
> expose all that complexity and advertise it.  We should try finding
> easier, more self-explanatory, customization means and options.

> Emacs 28 is the first
> release to include a decent support for Emoji, and we are still
> collecting user experience about what we have.


Agreed.  Fundamentally I think some script or other collection
mechanism that covers "emoji" is what's needed.  I think what's
in Emacs 28 is trying to do that and at least for this mac user
still falls short.

> NEWS describes what we think is the correct setup for Emoji.  I'm
> still not convinced that telling users to customize the font for
> 'symbol' is a good idea, especially since we prefer symbols and
> punctuation to be displayed by the default font as much as possible.

I'm still not clear on this.  With the set-fontset-font calls I describe I
see punctuation (; . ! etc) in Menlo (the font I put in the default and
fixed-pitch faces).  I assume this is because use-default-font-for-symbols
is t which is the default.  So at least on my system this preference works
out just as desired.

> Maybe this contradicts what macOS users expect, but macOS is not the
> main platform we want to support.

Understood but it's still a platform you do support.

> As for mentioning the fonts for which we set up by default -- there
> are others in the default fontset, and I don't see why those two are
> to be treated specially.  OTOH, mentioning all of them would be too
> much, because there are many, and they differ by platforms.  We should
> instead strive to make the default setup work seamlessly for everyone,
> IMO.

Agreed, but the mac platform is confounded because you can't
mention or pre-configure the default font to be used in this case, 
which is fine.  So all I'm asking for is some documentation someplace
that would lead a mac user to the correct incantation or some description
of the possible limitations encountered.  In the absence of "easier, 
more self-explanatory, customization means and options" this seems
like the most reasonable stopgap.

Howard

bug#54970: 28.1; Some emoji no longer display

[Eli Zaretskii]

> From: Howard Melman <hmelman@gmail.com>
> Date: Sat, 23 Apr 2022 13:22:45 -0400
> Cc: 54970@debbugs.gnu.org
> 
> > NEWS describes what we think is the correct setup for Emoji.  I'm
> > still not convinced that telling users to customize the font for
> > 'symbol' is a good idea, especially since we prefer symbols and
> > punctuation to be displayed by the default font as much as possible.
> 
> I'm still not clear on this.  With the set-fontset-font calls I describe I
> see punctuation (; . ! etc) in Menlo (the font I put in the default and
> fixed-pitch faces).  I assume this is because use-default-font-for-symbols
> is t which is the default.  So at least on my system this preference works
> out just as desired.

Yes, but the problematic cases is when the symbols/punctuation
coverage of the default font and of the font set up for the Symbols
and punctuation blocks overlap, and overlap significantly.

> > As for mentioning the fonts for which we set up by default -- there
> > are others in the default fontset, and I don't see why those two are
> > to be treated specially.  OTOH, mentioning all of them would be too
> > much, because there are many, and they differ by platforms.  We should
> > instead strive to make the default setup work seamlessly for everyone,
> > IMO.
> 
> Agreed, but the mac platform is confounded because you can't
> mention or pre-configure the default font to be used in this case, 
> which is fine.  So all I'm asking for is some documentation someplace
> that would lead a mac user to the correct incantation or some description
> of the possible limitations encountered.  In the absence of "easier, 
> more self-explanatory, customization means and options" this seems
> like the most reasonable stopgap.

We did that in NEWS, didn't we?

I'd prefer to see whether this is a problem for enough users for us to
do something else about it except answering questions.  If it turns
out we do need that, my first thought would be to encourage people to
use describe-fontset to see what they have, and take it from there.

bug#54970: 28.1; Some emoji no longer display

[Howard Melman]

> On Apr 23, 2022, at 1:46 PM, Eli Zaretskii <eliz@gnu.org> wrote:

> Yes, but the problematic cases is when the symbols/punctuation
> coverage of the default font and of the font set up for the Symbols
> and punctuation blocks overlap, and overlap significantly.
> 
>> Agreed, but the mac platform is confounded because you can't
>> mention or pre-configure the default font to be used in this case, 
>> which is fine.  So all I'm asking for is some documentation someplace
>> that would lead a mac user to the correct incantation or some description
>> of the possible limitations encountered.  In the absence of "easier, 
>> more self-explanatory, customization means and options" this seems
>> like the most reasonable stopgap.
> 
> We did that in NEWS, didn't we?

Well you just did it for script 'emoji and that didn't cover everything
for me.  Perhaps most of the issues were the sequences that didn't
include 0xFE0F and it would be great if there were a simple way to
define all of those in a fontset or if the emoji input stuff on master
was available for Emacs 28. There are some others, I think older
emoji that are problematic.  

I think also that Apple Color Emoji doesn't define glyphs for the
majority of "problematic" symbols so putting it in the fontset
for those codepoints isn't a problem and solves the missing
0xFE0F issue.

> I'd prefer to see whether this is a problem for enough users for us to
> do something else about it except answering questions.  If it turns
> out we do need that, my first thought would be to encourage people to
> use describe-fontset to see what they have, and take it from there.

I'll keep playing with it.  Maybe I'm the only emacs user on mac
that uses emoji.

Howard