bug#57141: 29.0.50; Most emoji not displaying in PGTK emacs

bug#57141: 29.0.50; Most emoji not displaying in PGTK emacs

[Michael Eliachevitch]

I use emacs under wayland (sway) with PGTK and native compilation,
compiled from master. However, I found that in this configuration, at
least in my recent builds, unicode emoji are not properly displayed.
E.g. if I call `emoji-list', most emoji are not shown at all, except
very basic emoji like "Smileys > affection." If I insert a unicode
emoji, I just see a unicode placeholder. Calling `describe-char' on the
"Smileys > smiling" emoji from the *Emoji* buffer gives gives me:

                position: 65 of 2529 (3%), column: 0
                character: C-j (displayed as C-j) (codepoint 10, #o12, #xa)
                charset: ascii (ASCII (ISO646 IRV))
    code point in charset: 0x0A
                script: latin
                syntax:   	which means: whitespace
                to input: type "C-x 8 RET a" or "C-x 8 RET LINE FEED (LF)"
            buffer code: #x0A
                file code: not encodable by coding system nil
                display: no font available

I tried using a different emoji font via customizing the emoji face,
where I set the font family to "JoyPixels", but this resulted in emacs crashing.

I could not find an existing ticket with "emoji" in the subject that
fits my symptoms, but maybe I didn't look properly or this is a symptom
of another known bug. Or I just misconfigured something on my system
which results in problems with the coding system or something that
happens due to my compilation options. Anyway, I don't know what else to
do to fix this on my own, so I would be happy for hints of if this can
be fixed somehow.

Best regards,
Michael Eliachevitch
---

In GNU Emacs 29.0.50 (build 2, x86_64-pc-linux-gnu, GTK+ Version 3.24.34, cairo version 1.17.6)
 of 2022-08-11 built on e490
Repository revision: 89f51673792b13ae0d1b93d0bf8e35d452693c9c
Repository branch: makepkg
System Description: Arch Linux

Configured using:
 'configure --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib
 --localstatedir=/var --mandir=/usr/share/man --with-gameuser=:games
 --with-modules --without-libotf --without-m17n-flt --without-gconf
 --enable-link-time-optimization --with-native-compilation
 --with-xinput2 --with-pgtk --without-xaw3d --with-sound=no
 --with-xwidgets --without-gpm --without-compress-install
 '--program-transform-name=s/\([ec]tags\)/\1.emacs/'
 'CFLAGS=-march=native -mtune=generic -O3 -pipe -fno-plt -fexceptions
 -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security
 -fstack-clash-protection -fcf-protection'
 LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG JSON
LCMS2 LIBSYSTEMD LIBXML2 MODULES NATIVE_COMP NOTIFY INOTIFY PDUMPER PGTK
PNG RSVG SECCOMP SQLITE3 THREADS TIFF TOOLKIT_SCROLL_BARS WEBP XIM
XWIDGETS GTK3 ZLIB

Important settings:
  value of $LC_ALL: C
  value of $LC_COLLATE: C.UTF-8
  value of $LC_CTYPE: en_US.UTF-8
  value of $LC_MESSAGES: en_US.UTF-8
  value of $LC_MONETARY: de_DE.UTF-8
  value of $LC_NUMERIC: en_US.UTF-8
  value of $LC_TIME: de_DE.UTF-8
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=fcitx
  locale-coding-system: nil

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message mailcap yank-media puny dired
dired-loaddefs rfc822 mml mml-sec password-cache epa derived epg rfc6068
epg-config gnus-util text-property-search time-date mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader sendmail
rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils emoji-labels
emoji generate-lisp-file multisession sqlite transient comp comp-cstr
warnings icons subr-x rx cl-seq format-spec edmacro kmacro eieio
byte-opt bytecomp byte-compile cconv eieio-core cl-macs gv cl-extra
help-mode cl-loaddefs cl-lib rmc iso-transl tooltip eldoc paren electric
uniquify ediff-hook vc-hooks lisp-float-type elisp-mode mwheel
term/pgtk-win pgtk-win term/common-win pgtk-dnd tool-bar dnd fontset
image regexp-opt fringe tabulated-list replace newcomment text-mode
lisp-mode prog-mode register page tab-bar menu-bar rfn-eshadow isearch
easymenu timer select scroll-bar mouse jit-lock font-lock syntax
font-core term/tty-colors frame minibuffer nadvice seq simple cl-generic
indonesian philippine cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
composite emoji-zwj charscript charprop case-table epa-hook
jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button loaddefs
faces cus-face macroexp files window text-properties overlay sha1 md5
base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads xwidget-internal dbusbind
inotify dynamic-setting system-font-setting font-render-setting cairo
gtk pgtk lcms2 multi-tty make-network-process native-compile emacs)

Memory information:
((conses 16 118945 18632)
 (symbols 48 21096 1)
 (strings 32 102059 1853)
 (string-bytes 1 2486759)
 (vectors 16 26117)
 (vector-slots 8 1148486 18229)
 (floats 8 39 31)
 (intervals 56 744 0)
 (buffers 992 13))

--
Michael Eliachevitch
Public PGP Key: https://keyoxide.org/hkp/546908c782383ad0e7d894ec1b8f95c8125dce31

bug#57141: 29.0.50; Most emoji not displaying in PGTK emacs

[Eli Zaretskii]

> From: Michael Eliachevitch <m.eliachevitch@posteo.de>
> Date: Thu, 11 Aug 2022 14:10:25 +0000
> 
> I use emacs under wayland (sway) with PGTK and native compilation,
> compiled from master. However, I found that in this configuration, at
> least in my recent builds, unicode emoji are not properly displayed.
> E.g. if I call `emoji-list', most emoji are not shown at all, except
> very basic emoji like "Smileys > affection." If I insert a unicode
> emoji, I just see a unicode placeholder. Calling `describe-char' on the
> "Smileys > smiling" emoji from the *Emoji* buffer gives gives me:

Could this be bug#57066?  Does the patch posted there fix the problem?

>                 position: 65 of 2529 (3%), column: 0
>                 character: C-j (displayed as C-j) (codepoint 10, #o12, #xa)
>                 charset: ascii (ASCII (ISO646 IRV))
>     code point in charset: 0x0A
>                 script: latin
>                 syntax:   	which means: whitespace
>                 to input: type "C-x 8 RET a" or "C-x 8 RET LINE FEED (LF)"
>             buffer code: #x0A
>                 file code: not encodable by coding system nil
>                 display: no font available

This is not an Emoji character, this is NBSP.

Do you have Color Note Emoji font installed?

bug#57141: 29.0.50; Most emoji not displaying in PGTK emacs

[Michael Eliachevitch]

[-- Attachment #1: Type: text/plain, Size: 3200 bytes --]

> This is not an Emoji character, this is NBSP.

Upps, my bad, obviously, seems that `(emoji-list)' skips those unrenderable emoji.
For describe-char on "😊", I get

--8<---------------cut here---------------start------------->8---
                position: 523 of 1964 (27%), restriction: <103-1965>, column: 0
                character: 😊 (displayed as 😊) (codepoint 128522, #o373012, #x1f60a)
                charset: unicode (Unicode (ISO10646))
    code point in charset: 0x1F60A
                script: emoji
                syntax: w 	which means: word
                category: .:Base
                to input: type "C-x 8 RET 1f60a" or "C-x 8 RET SMILING FACE WITH SMILING EYES"
            buffer code: #xF0 #x9F #x98 #x8A
                file code: #xF0 #x9F #x98 #x8A (encoded by coding system utf-8-unix)
                display: no font available

    Character code properties: customize what to show
    name: SMILING FACE WITH SMILING EYES
    general-category: So (Symbol, Other)
    decomposition: (128522) ('😊')

    There are text properties here:
    fontified            t
    wrap-prefix          ""
    ws-butler-chg        chg
--8<---------------cut here---------------end--------------->8---

I now also attached some screenshots of the emoji-list and also of the completion-buffer when I press <tab> in emoji-search, which shows the unicode placeholders that I get, so it might be more clear what my problem is.

> Do you have Color Note Emoji font installed?

 "Noto Color Emoji" is in the output of '(font-family-list) ` and it's also installed via my system package manager. Weirdly, if try to change the frame font to "Noto Color Emoji", which I did via the menu-bar "options->set default font", I get the error message

    set-frame-font: Font not available: #<font-spec nil nil Noto\ Color\ Emoji nil nil normal normal nil 10.0 nil nil nil nil>

I get this for all other emoji fonts that I have installed, but maybe this is normal for an emoji font because setting it as font for the entire frame doesn't make much sense, but I don't know if this is not a hint of an underlying font-config issue.

> Could this be bug#57066?  Does the patch posted there fix the problem?

I agree this is likely to be an issue with harfbuzz, even though the symptoms of this bug look different from bug#57066, as I just don't get any rendered emoji instead of overlapping emoji (see attached screenshots). I will try this patch anyway after this mail.

I also saw bug#56789 which seemed similar in symptoms, but I just checked that my archlinux installation has harfbuzz 5.1.0, where the bug causing #56789 is should be fixed as far as I understood. I tried a clean rebuild to make sure I use that harfbuzz version and also I tried downgrading harfbuzz to 4.4.1 and rebuilding emacs and neither solved my rendering issues. I also tried the command

    hb-view  --output-file=foo.svg --font-size=13 \
    /usr/share/fonts/noto/NotoColorEmoji.ttf \
    -u 1f469,200d,2764,fe0f,200d,1f468

and it works, producing a nice SVG emoji image. So it's not that harfbuzz in general is broken.

Thanks for the help,
Michael



[-- Attachment #2: emoji-list.png --]
[-- Type: image/png, Size: 28930 bytes --]

[-- Attachment #3: emoji-completion.png --]
[-- Type: image/png, Size: 21634 bytes --]

bug#57141: 29.0.50; Most emoji not displaying in PGTK emacs

[Eli Zaretskii]

> From: Michael Eliachevitch <m.eliachevitch@posteo.de>
> Cc: 57141@debbugs.gnu.org
> Date: Thu, 11 Aug 2022 17:13:32 +0000
> 
>                 position: 523 of 1964 (27%), restriction: <103-1965>, column: 0
>                 character: 😊 (displayed as 😊) (codepoint 128522, #o373012, #x1f60a)
>                 charset: unicode (Unicode (ISO10646))
>     code point in charset: 0x1F60A
>                 script: emoji
>                 syntax: w 	which means: word
>                 category: .:Base
>                 to input: type "C-x 8 RET 1f60a" or "C-x 8 RET SMILING FACE WITH SMILING EYES"
>             buffer code: #xF0 #x9F #x98 #x8A
>                 file code: #xF0 #x9F #x98 #x8A (encoded by coding system utf-8-unix)
>                 display: no font available
                           ^^^^^^^^^^^^^^^^^
This means Emacs cannot find a font for this character.  You need to
figure out why.

> > Do you have Color Note Emoji font installed?
> 
>  "Noto Color Emoji" is in the output of '(font-family-list) ` and it's also installed via my system package manager. Weirdly, if try to change the frame font to "Noto Color Emoji", which I did via the menu-bar "options->set default font", I get the error message
> 
>     set-frame-font: Font not available: #<font-spec nil nil Noto\ Color\ Emoji nil nil normal normal nil 10.0 nil nil nil nil>

This again means Emacs cannot use that font, for some reason.

bug#57141: 29.0.50; Most emoji not displaying in PGTK emacs

[Michael Eliachevitch]

[-- Attachment #1: Type: text/plain, Size: 867 bytes --]

I figured out this issue, it was due to my custom font configuration in ~/.config/fontconfig/fonts.conf'. When deleting that file, emojis display and I can also set the face- and frame-fonts to emoji fonts. Probably it's just broken, but I think it's still possible that it's formally correct, but the hinting and aliasing rules that it specifies cause some bug down the line. Feel free to close this, though if anyone is interested, I attached the file.

I also tried applying the suggested patch from bug#57066, but saw no difference: Without the custom fonts.conf the emoji display works well whether the patch is applied or not . With the custom fonts.conf the patch doesn't fix the issue.

Thanks so much for the help Eli! I'm glad to have this finally solved as I'm seeing more and more emacs users and packages starting to use unicode emoji.

Cheers, Michael
 

[-- Attachment #2: fonts.conf --]
[-- Type: text/plain, Size: 907 bytes --]

<?xml version='1.0' encoding='UTF-8' standalone='no'?>
<fontconfig>
    <dir>~/.fonts</dir>
    <match target="font">
        <edit name="rgba" mode="assign">
            <const>none</const>
        </edit>
    </match>
    <match target="font">
        <edit name="hinting" mode="assign">
            <bool>true</bool>
        </edit>
    </match>
    <match target="font">
        <edit name="hintstyle" mode="assign">
            <const>hintfull</const>
        </edit>
    </match>
    <match target="font">
        <edit name="antialias" mode="assign">
            <bool>true</bool>
        </edit>
    </match>
    <match target="font">
        <edit mode="assign" name="lcdfilter">
            <const>lcddefault</const>
        </edit>
    </match>
    <match target="font">
        <edit mode="assign" name="embeddedbitmap">
            <bool>false</bool>
        </edit>
    </match>
</fontconfig>

bug#57141: 29.0.50; Most emoji not displaying in PGTK emacs

[Eli Zaretskii]

> From: Michael Eliachevitch <m.eliachevitch@posteo.de>
> Cc: 57141@debbugs.gnu.org
> Date: Thu, 11 Aug 2022 18:54:29 +0000
> 
> I figured out this issue, it was due to my custom font configuration in ~/.config/fontconfig/fonts.conf'. When deleting that file, emojis display and I can also set the face- and frame-fonts to emoji fonts. Probably it's just broken, but I think it's still possible that it's formally correct, but the hinting and aliasing rules that it specifies cause some bug down the line. Feel free to close this, though if anyone is interested, I attached the file.
> 
> I also tried applying the suggested patch from bug#57066, but saw no difference: Without the custom fonts.conf the emoji display works well whether the patch is applied or not . With the custom fonts.conf the patch doesn't fix the issue.
> 
> Thanks so much for the help Eli! I'm glad to have this finally solved as I'm seeing more and more emacs users and packages starting to use unicode emoji.

Thank you for telling us.  I'm therefore closing this bug.