bug#54441: 28.0.92; set-language-environment causes Apple Color Emoji cannot show up

bug#54441: 28.0.92; set-language-environment causes Apple Color Emoji cannot show up

[Will Chang]

[-- Attachment #1: Type: text/plain, Size: 3654 bytes --]

This bug shows up with the code following:

;; 😄

(set-fontset-font "fontset-default" 'unicode "Apple Color Emoji" nil
'prepend)

(set-language-environment "UTF-8")

Steps to reproduce:
1. eval-buffer and the emoji doesn't show up correctly.
2. Comment out (set-language-environment "UTF-8") and run eval-buffer
the emoji shows up correctly.


I encountered this bug from Emacs mac port
bitbucket.org/mituharu/emacs-mac/src/work/ 28.0.92 to make sure it's
not only on that port, I download Emacs from emacsformacosx.com and
reproduced the bug. I found that emacs -Q is not available in Terminal so I
tested
it with no init files.

This is the first time I reported a bug to Emacs, please forgive me for any
inappropriate thing.



In GNU Emacs 28.0.92 (build 1, x86_64-apple-darwin18.7.0, NS appkit-1671.60
Version 10.14.6 (Build 18G95))
 of 2022-03-14 built on builder10-14.lan
Windowing system distributor 'Apple', version 10.3.2113
System Description:  macOS 12.3

Configured using:
 'configure --with-ns '--enable-locallisppath=/Library/Application
 Support/Emacs/${version}/site-lisp:/Library/Application
 Support/Emacs/site-lisp' --with-modules'

Configured features:
ACL GMP GNUTLS JSON LIBXML2 MODULES NOTIFY KQUEUE NS PDUMPER THREADS
TOOLKIT_SCROLL_BARS ZLIB

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: ELisp/d

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message rmc puny dired dired-loaddefs
rfc822 mml mml-sec epa derived epg rfc6068 epg-config gnus-util rmail
rmail-loaddefs auth-source cl-seq eieio eieio-core cl-macs
eieio-loaddefs password-cache json map text-property-search time-date
subr-x seq byte-opt gv bytecomp byte-compile cconv mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader cl-loaddefs
cl-lib sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils
time iso-transl tooltip eldoc paren electric uniquify ediff-hook
vc-hooks lisp-float-type elisp-mode mwheel term/ns-win ns-win
ucs-normalize mule-util term/common-win tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu
timer select scroll-bar mouse jit-lock font-lock syntax font-core
term/tty-colors frame minibuffer cl-generic cham georgian utf-8-lang
misc-lang vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms
cp51932 hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese composite emoji-zwj charscript charprop case-table
epa-hook jka-cmpr-hook help simple abbrev obarray cl-preloaded nadvice
button loaddefs faces cus-face macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget
hashtable-print-readable backquote threads kqueue cocoa ns multi-tty
make-network-process emacs)

Memory information:
((conses 16 54262 10538)
 (symbols 48 6616 1)
 (strings 32 18521 1412)
 (string-bytes 1 615366)
 (vectors 16 14366)
 (vector-slots 8 197125 10303)
 (floats 8 23 40)
 (intervals 56 237 2)
 (buffers 992 13))

[-- Attachment #2: Type: text/html, Size: 4137 bytes --]

bug#54441: 28.0.92; set-language-environment causes Apple Color Emoji cannot show up

[Eli Zaretskii]

> From: Will Chang <will.b.chang@gmail.com>
> Date: Fri, 18 Mar 2022 11:41:00 +0800
> 
> (set-fontset-font "fontset-default" 'unicode "Apple Color Emoji" nil 'prepend)
> 
> (set-language-environment "UTF-8")
> 
> Steps to reproduce:
> 1. eval-buffer and the emoji doesn't show up correctly.
> 2. Comment out (set-language-environment "UTF-8") and run eval-buffer
> the emoji shows up correctly.

Please tell:

  . does this happen in "emacs -Q", if you evaluate the above
    immediately upon entering Emacs?
  . what is the language environment before the evaluation of
    set-language-environment above?
  . what is the output of "M-x describe-fontset RET fontset-default RET"
    after evaluating set-language-environment, in particular in the
    Emoji block starting from #x1F600?
  . what did you mean by "emoji doesn't show up correctly"? how was it
    displayed "incorrectly"?

And I have 2 comments, which may or may not be relevant to this issue:

 1) It is a very bad idea to do the likes of

    (set-fontset-font "fontset-default" 'unicode "Apple Color Emoji" nil 'prepend)

    That's because the character set 'unicode spans the entire range
    of Unicode characters, and there's no single font that can
    possibly support all of those characters.

 2) You should set the language environment _before_ any other
    customizations related to character sets and fonts, because the
    language environment affects those customizations.

Thanks.

bug#54441: 28.0.92; set-language-environment causes Apple Color Emoji cannot show up

[Eli Zaretskii]

[Please use Reply All so that the bug tracker is CC'ed.]

> From: Will Chang <will.b.chang@gmail.com>
> Date: Sat, 19 Mar 2022 11:06:06 +0800
> 
> >does this happen in "emacs -Q", if you evaluate the above
> immediately upon entering Emacs?
> 
> Yes. Tested it on:
> "GNU Emacs 28.0.92 (build 2, x86_64-apple-darwin21.4.0, Carbon Version 165
> AppKit 2113.4)
>  of 2022-03-18"
> "GNU Emacs 28.0.92 (build 1, x86_64-apple-darwin18.7.0, NS appkit-1671.60
> Version 10.14.6 (Build 18G95))
>  of 2022-03-14"
> 
> > what is the language environment before the evaluation of
>     set-language-environment above?
> 
> I run describe-language-environment and got:
> 
> English language environment
> 
> Nothing special is needed to handle English.
> 
> Sample text:
>   Hello!, Hi!, How are you?
> 
> Input methods:
>   programmer-dvorak ("DVP@" in mode line)
>   english-dvorak ("DV@" in mode line)
> 
> Character sets:
>   ascii: ASCII (ISO646 IRV)
> 
> Coding systems:
>   nothing specific to English

So it's a "C" (a.k.a. "Posix") language environment, looks like.
Somewhat strange, but maybe this is "usual" on macOS?

> > what is the output of "M-x describe-fontset RET fontset-default RET"
>     after evaluating set-language-environment, in particular in the
>     Emoji block starting from #x1F600?
> 
> 😀 .. 🙏 (#x1F600 .. #x1F64F)
>     -*-Apple Color Emoji-*-*-*-*-*-*-*-*-*-*-*-*
>     -*-Noto Color Emoji-*-*-*-*-*-*-*-*-*-*-iso10646-1
>     -*-Symbola-*-*-*-*-*-*-*-*-*-*-iso10646-1
> [-*-Symbola-normal-normal-semicondensed-*-12-*-*-*-p-0-iso10646-1]

It sounds like Emacs used Symbola, although Apple Color Emoji is in
the list, and is first in the list.  Which probably means Emacs
examined Apple Color Emoji and rejected it for some reason.

What happens if you evaluate

  (set-fontset-font "fontset-default" 'unicode "Apple Color Emoji" nil 'prepend)

again, after changing the language environment?

> >what did you mean by "emoji doesn't show up correctly"? how was it
>     displayed "incorrectly"?
> 
> The screenshots:
> - default: https://i.imgur.com/jLcwEDk.png
> - English with fontset(correctly): https://i.imgur.com/qgqhCHF.png
> - UTF-8 with fontset: https://i.imgur.com/tkjOVGv.png

AFAICT, this shows that "incorrect" means Emacs uses the Symbola font.

Does Apple Color Emoji on your system have glyphs for the #x1F300 and
#x1F600 characters?  Those are the characters we request to be
supported by a font that is used for emoji.

bug#54441: 28.0.92; set-language-environment causes Apple Color Emoji cannot show up

[Will Chang]

[-- Attachment #1: Type: text/plain, Size: 3876 bytes --]

> So it's a "C" (a.k.a. "Posix") language environment, looks like.
> Somewhat strange, but maybe this is "usual" on macOS?

On the previous Emacs Mac Port versions, Emacs doesn't need to set
the code below to enable Emoji and the UTF-8 environment doesn't affect it.

  (set-fontset-font "fontset-default" 'unicode "Apple Color Emoji" nil
'prepend)

Maybe it's a bug on that port, I CC'ed the maintainer Mituharu.


> What happens if you evaluate
>
>   (set-fontset-font "fontset-default" 'unicode "Apple Color Emoji" nil
'prepend)
>
> again, after changing the language environment?

It remains the same.

> AFAICT, this shows that "incorrect" means Emacs uses the Symbola font.

Thanks for the info.

> Does Apple Color Emoji on your system have glyphs for the #x1F300 and
> #x1F600 characters?  Those are the characters we request to be
> supported by a font that is used for emoji.

I believe it has.


Currently, I removed the UTF-8 code and set the Apple Color Emoji font for a
temporary solution.

Again, thanks for your help, I learned a lot with your conversation, you're
so kind. ;)

On Sat, Mar 19, 2022 at 2:53 PM Eli Zaretskii <eliz@gnu.org> wrote:

> [Please use Reply All so that the bug tracker is CC'ed.]
>
> > From: Will Chang <will.b.chang@gmail.com>
> > Date: Sat, 19 Mar 2022 11:06:06 +0800
> >
> > >does this happen in "emacs -Q", if you evaluate the above
> > immediately upon entering Emacs?
> >
> > Yes. Tested it on:
> > "GNU Emacs 28.0.92 (build 2, x86_64-apple-darwin21.4.0, Carbon Version
> 165
> > AppKit 2113.4)
> >  of 2022-03-18"
> > "GNU Emacs 28.0.92 (build 1, x86_64-apple-darwin18.7.0, NS appkit-1671.60
> > Version 10.14.6 (Build 18G95))
> >  of 2022-03-14"
> >
> > > what is the language environment before the evaluation of
> >     set-language-environment above?
> >
> > I run describe-language-environment and got:
> >
> > English language environment
> >
> > Nothing special is needed to handle English.
> >
> > Sample text:
> >   Hello!, Hi!, How are you?
> >
> > Input methods:
> >   programmer-dvorak ("DVP@" in mode line)
> >   english-dvorak ("DV@" in mode line)
> >
> > Character sets:
> >   ascii: ASCII (ISO646 IRV)
> >
> > Coding systems:
> >   nothing specific to English
>
> So it's a "C" (a.k.a. "Posix") language environment, looks like.
> Somewhat strange, but maybe this is "usual" on macOS?
>
> > > what is the output of "M-x describe-fontset RET fontset-default RET"
> >     after evaluating set-language-environment, in particular in the
> >     Emoji block starting from #x1F600?
> >
> > 😀 .. 🙏 (#x1F600 .. #x1F64F)
> >     -*-Apple Color Emoji-*-*-*-*-*-*-*-*-*-*-*-*
> >     -*-Noto Color Emoji-*-*-*-*-*-*-*-*-*-*-iso10646-1
> >     -*-Symbola-*-*-*-*-*-*-*-*-*-*-iso10646-1
> > [-*-Symbola-normal-normal-semicondensed-*-12-*-*-*-p-0-iso10646-1]
>
> It sounds like Emacs used Symbola, although Apple Color Emoji is in
> the list, and is first in the list.  Which probably means Emacs
> examined Apple Color Emoji and rejected it for some reason.
>
> What happens if you evaluate
>
>   (set-fontset-font "fontset-default" 'unicode "Apple Color Emoji" nil
> 'prepend)
>
> again, after changing the language environment?
>
> > >what did you mean by "emoji doesn't show up correctly"? how was it
> >     displayed "incorrectly"?
> >
> > The screenshots:
> > - default: https://i.imgur.com/jLcwEDk.png
> > - English with fontset(correctly): https://i.imgur.com/qgqhCHF.png
> > - UTF-8 with fontset: https://i.imgur.com/tkjOVGv.png
>
> AFAICT, this shows that "incorrect" means Emacs uses the Symbola font.
>
> Does Apple Color Emoji on your system have glyphs for the #x1F300 and
> #x1F600 characters?  Those are the characters we request to be
> supported by a font that is used for emoji.
>

[-- Attachment #2: Type: text/html, Size: 5848 bytes --]

bug#54441: 28.0.92; set-language-environment causes Apple Color Emoji cannot show up

[Will Chang]

[-- Attachment #1: Type: text/plain, Size: 4913 bytes --]

Hi Eli,

I searched this problem again and found:
https://www.reddit.com/r/emacs/comments/pmghug/how_to_correctly_display_multicharacter_emojis/
https://lists.gnu.org/archive/html/emacs-devel/2021-09/msg01238.html

The code below can solve the problem.

  (set-fontset-font t 'emoji '("Apple Color Emoji" . "iso10646-1") nil
'prepend)

or

  (set-fontset-font t '(#x1f000 . #x1faff) (font-spec :family "Apple Color
Emoji"))


You are absolutely right on this:

> 1) It is a very bad idea to do the likes of
>
>    (set-fontset-font "fontset-default" 'unicode "Apple Color Emoji" nil
'prepend)
>
>    That's because the character set 'unicode spans the entire range
>    of Unicode characters, and there's no single font that can
>    possibly support all of those characters.


On Sat, Mar 19, 2022 at 4:51 PM Will Chang <will.b.chang@gmail.com> wrote:

> > So it's a "C" (a.k.a. "Posix") language environment, looks like.
> > Somewhat strange, but maybe this is "usual" on macOS?
>
> On the previous Emacs Mac Port versions, Emacs doesn't need to set
> the code below to enable Emoji and the UTF-8 environment doesn't affect it.
>
>   (set-fontset-font "fontset-default" 'unicode "Apple Color Emoji" nil
> 'prepend)
>
> Maybe it's a bug on that port, I CC'ed the maintainer Mituharu.
>
>
> > What happens if you evaluate
> >
> >   (set-fontset-font "fontset-default" 'unicode "Apple Color Emoji" nil
> 'prepend)
> >
> > again, after changing the language environment?
>
> It remains the same.
>
> > AFAICT, this shows that "incorrect" means Emacs uses the Symbola font.
>
> Thanks for the info.
>
> > Does Apple Color Emoji on your system have glyphs for the #x1F300 and
> > #x1F600 characters?  Those are the characters we request to be
> > supported by a font that is used for emoji.
>
> I believe it has.
>
>
> Currently, I removed the UTF-8 code and set the Apple Color Emoji font for
> a
> temporary solution.
>
> Again, thanks for your help, I learned a lot with your conversation,
> you're so kind. ;)
>
> On Sat, Mar 19, 2022 at 2:53 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
>> [Please use Reply All so that the bug tracker is CC'ed.]
>>
>> > From: Will Chang <will.b.chang@gmail.com>
>> > Date: Sat, 19 Mar 2022 11:06:06 +0800
>> >
>> > >does this happen in "emacs -Q", if you evaluate the above
>> > immediately upon entering Emacs?
>> >
>> > Yes. Tested it on:
>> > "GNU Emacs 28.0.92 (build 2, x86_64-apple-darwin21.4.0, Carbon Version
>> 165
>> > AppKit 2113.4)
>> >  of 2022-03-18"
>> > "GNU Emacs 28.0.92 (build 1, x86_64-apple-darwin18.7.0, NS
>> appkit-1671.60
>> > Version 10.14.6 (Build 18G95))
>> >  of 2022-03-14"
>> >
>> > > what is the language environment before the evaluation of
>> >     set-language-environment above?
>> >
>> > I run describe-language-environment and got:
>> >
>> > English language environment
>> >
>> > Nothing special is needed to handle English.
>> >
>> > Sample text:
>> >   Hello!, Hi!, How are you?
>> >
>> > Input methods:
>> >   programmer-dvorak ("DVP@" in mode line)
>> >   english-dvorak ("DV@" in mode line)
>> >
>> > Character sets:
>> >   ascii: ASCII (ISO646 IRV)
>> >
>> > Coding systems:
>> >   nothing specific to English
>>
>> So it's a "C" (a.k.a. "Posix") language environment, looks like.
>> Somewhat strange, but maybe this is "usual" on macOS?
>>
>> > > what is the output of "M-x describe-fontset RET fontset-default RET"
>> >     after evaluating set-language-environment, in particular in the
>> >     Emoji block starting from #x1F600?
>> >
>> > 😀 .. 🙏 (#x1F600 .. #x1F64F)
>> >     -*-Apple Color Emoji-*-*-*-*-*-*-*-*-*-*-*-*
>> >     -*-Noto Color Emoji-*-*-*-*-*-*-*-*-*-*-iso10646-1
>> >     -*-Symbola-*-*-*-*-*-*-*-*-*-*-iso10646-1
>> > [-*-Symbola-normal-normal-semicondensed-*-12-*-*-*-p-0-iso10646-1]
>>
>> It sounds like Emacs used Symbola, although Apple Color Emoji is in
>> the list, and is first in the list.  Which probably means Emacs
>> examined Apple Color Emoji and rejected it for some reason.
>>
>> What happens if you evaluate
>>
>>   (set-fontset-font "fontset-default" 'unicode "Apple Color Emoji" nil
>> 'prepend)
>>
>> again, after changing the language environment?
>>
>> > >what did you mean by "emoji doesn't show up correctly"? how was it
>> >     displayed "incorrectly"?
>> >
>> > The screenshots:
>> > - default: https://i.imgur.com/jLcwEDk.png
>> > - English with fontset(correctly): https://i.imgur.com/qgqhCHF.png
>> > - UTF-8 with fontset: https://i.imgur.com/tkjOVGv.png
>>
>> AFAICT, this shows that "incorrect" means Emacs uses the Symbola font.
>>
>> Does Apple Color Emoji on your system have glyphs for the #x1F300 and
>> #x1F600 characters?  Those are the characters we request to be
>> supported by a font that is used for emoji.
>>
>

[-- Attachment #2: Type: text/html, Size: 7674 bytes --]

bug#54441: 28.0.92; set-language-environment causes Apple Color Emoji cannot show up

[Eli Zaretskii]

> From: Will Chang <will.b.chang@gmail.com>
> Date: Sat, 19 Mar 2022 17:24:17 +0800
> Cc: 54441@debbugs.gnu.org, mituharu+bug-gnu-emacs-mac@math.s.chiba-u.ac.jp
> 
> I searched this problem again and found:
> https://www.reddit.com/r/emacs/comments/pmghug/how_to_correctly_display_multicharacter_emojis/
> https://lists.gnu.org/archive/html/emacs-devel/2021-09/msg01238.html
> 
> The code below can solve the problem.
> 
>   (set-fontset-font t 'emoji '("Apple Color Emoji" . "iso10646-1") nil 'prepend)
> 
> or
> 
>   (set-fontset-font t '(#x1f000 . #x1faff) (font-spec :family "Apple Color Emoji"))

So if you use the above instead of your original set-fontset-font, the
problem disappears, and the emoji are displayed correctly even after
set-language-environment, is that right?

bug#54441: 28.0.92; set-language-environment causes Apple Color Emoji cannot show up

[Will Chang]

[-- Attachment #1: Type: text/plain, Size: 960 bytes --]

Yes, it works well even after set-language-environment to UTF-8.

On Sat, Mar 19, 2022 at 5:33 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Will Chang <will.b.chang@gmail.com>
> > Date: Sat, 19 Mar 2022 17:24:17 +0800
> > Cc: 54441@debbugs.gnu.org,
> mituharu+bug-gnu-emacs-mac@math.s.chiba-u.ac.jp
> >
> > I searched this problem again and found:
> >
> https://www.reddit.com/r/emacs/comments/pmghug/how_to_correctly_display_multicharacter_emojis/
> > https://lists.gnu.org/archive/html/emacs-devel/2021-09/msg01238.html
> >
> > The code below can solve the problem.
> >
> >   (set-fontset-font t 'emoji '("Apple Color Emoji" . "iso10646-1") nil
> 'prepend)
> >
> > or
> >
> >   (set-fontset-font t '(#x1f000 . #x1faff) (font-spec :family "Apple
> Color Emoji"))
>
> So if you use the above instead of your original set-fontset-font, the
> problem disappears, and the emoji are displayed correctly even after
> set-language-environment, is that right?
>

[-- Attachment #2: Type: text/html, Size: 1850 bytes --]

bug#54441: 28.0.92; set-language-environment causes Apple Color Emoji cannot show up

[Eli Zaretskii]

> From: Will Chang <will.b.chang@gmail.com>
> Date: Sat, 19 Mar 2022 17:36:17 +0800
> Cc: 54441@debbugs.gnu.org, mituharu+bug-gnu-emacs-mac@math.s.chiba-u.ac.jp
> 
> Yes, it works well even after set-language-environment to UTF-8.

Great, thanks.  Then I think there's no Emacs bug here, and I'm
therefore closing this bug report.