bug#65993: 29.1; emoji-insert show nothing without font settings

bug#65993: 29.1; emoji-insert show nothing without font settings

[awrhygty]

Without font settings, almost all emojis are not displayed within emacs,
and emoji-insert does not show any leaf menu items.
For example, typing 'C-x 8 e e s s' (Emoji > Smileys > smiling)
emacs shows empty minibuffer(no string shown).

In such situation I can not understand what is going on.
Empty menu should show some information or raise an error.

And once emoji-insert is called before font setting,
emoji-insert does not show emoji items after they are made visible in
emacs buffer with font settings.

Here is a sample setting to display emojis:
  (set-fontset-font nil '(#x1f000 . #x1faff) "Segoe UI Emoji")
Without font settings, characters in the range are displayed with
  'Segoe UI Symbol'
    1F000..1F003, 1F005..1F02B, 1F030..1F093
  'ＭＳ ゴシック'(MS Gothic)
    1F200..1F202, 1F210..1F23A, 1F240..1F248, 1F250..1F251
  hex numbers
    other characters


In GNU Emacs 29.1 (build 2, x86_64-w64-mingw32) of 2023-08-02 built on
 AVALON
Windowing system distributor 'Microsoft Corp.', version 10.0.19045
System Description: Microsoft Windows 10 Pro (v10.0.2009.19045.3448)

Configured using:
 'configure --with-modules --without-dbus --with-native-compilation=aot
 --without-compress-install --with-tree-sitter CFLAGS=-O2'

Configured features:
ACL GIF GMP GNUTLS HARFBUZZ JPEG JSON LCMS2 LIBXML2 MODULES NATIVE_COMP
NOTIFY W32NOTIFY PDUMPER PNG RSVG SOUND SQLITE3 THREADS TIFF
TOOLKIT_SCROLL_BARS TREE_SITTER WEBP XPM ZLIB

(NATIVE_COMP present but libgccjit not available)

Important settings:
  value of $LANG: JPN
  locale-coding-system: cp932

Major mode: Lisp Interaction

Minor modes in effect:
  highlight-changes-visible-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
None found.

Features:
(gnutls network-stream nsm mailalias smtpmail textsec uni-scripts url
url-proxy url-privacy url-expand url-methods url-history url-cookie
generate-lisp-file url-domsuf url-util url-parse auth-source cl-seq
eieio eieio-core cl-macs json map url-vars idna-mapping ucs-normalize
uni-confusable textsec-check cl-print byte-opt gv bytecomp byte-compile
debug backtrace find-func hilit-chg wid-edit thingatpt help-fns
radix-tree help-mode pp shadow sort mail-extr emacsbug message mailcap
yank-media puny dired dired-loaddefs rfc822 mml mml-sec password-cache
epa derived epg rfc6068 epg-config gnus-util text-property-search
time-date subr-x mm-decode mm-bodies mm-encode mail-parse rfc2231
mailabbrev gmm-utils mailheader cl-loaddefs cl-lib sendmail rfc2047
rfc2045 ietf-drums mm-util mail-prsvr mail-utils term/bobcat japan-util
rmc iso-transl tooltip cconv eldoc paren electric uniquify ediff-hook
vc-hooks lisp-float-type elisp-mode mwheel dos-w32 ls-lisp disp-table
term/w32-win w32-win w32-vars term/common-win tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode lisp-mode
prog-mode register page tab-bar menu-bar rfn-eshadow isearch easymenu
timer select scroll-bar mouse jit-lock font-lock syntax font-core
term/tty-colors frame minibuffer nadvice seq simple cl-generic
indonesian philippine cham georgian utf-8-lang misc-lang vietnamese
tibetan thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek
romanian slovak czech european ethiopic indian cyrillic chinese
composite emoji-zwj charscript charprop case-table epa-hook
jka-cmpr-hook help abbrev obarray oclosure cl-preloaded button loaddefs
theme-loaddefs faces cus-face macroexp files window text-properties
overlay sha1 md5 base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads w32notify w32 lcms2 multi-tty
make-network-process native-compile emacs)

Memory information:
((conses 16 229737 11617)
 (symbols 48 8112 3)
 (strings 32 40380 2121)
 (string-bytes 1 1018919)
 (vectors 16 40562)
 (vector-slots 8 811973 35006)
 (floats 8 49 287)
 (intervals 56 776 0)
 (buffers 984 15))

bug#65993: 29.1; emoji-insert show nothing without font settings

[Eli Zaretskii]

merge 65993 65138
thanks

> From: awrhygty@outlook.com
> Date: Fri, 15 Sep 2023 21:06:18 +0900
> 
> 
> Without font settings, almost all emojis are not displayed within emacs,
> and emoji-insert does not show any leaf menu items.
> For example, typing 'C-x 8 e e s s' (Emoji > Smileys > smiling)
> emacs shows empty minibuffer(no string shown).
> 
> In such situation I can not understand what is going on.
> Empty menu should show some information or raise an error.

This is a duplicate of bug#65138, for which we don't yet have a
solution.  Patches welcome.

> And once emoji-insert is called before font setting,
> emoji-insert does not show emoji items after they are made visible in
> emacs buffer with font settings.

This is not a bug: you need to clear the font cache to get Emacs to
react to newly-installed fonts without restarting Emacs.  Like this:

  M-: (clear-font-cache) RET

> Here is a sample setting to display emojis:
>   (set-fontset-font nil '(#x1f000 . #x1faff) "Segoe UI Emoji")

Segoe UI Emoji is not a free font, so we cannot advertise it in Emacs.
You could tell Emacs about any fonts you have that support Emoji by
using the function w32-find-non-USB-fonts, see its doc string for
details.

In addition, Emacs 29 comes pre-configured to use the Noto Emoji font
on MS-Windows for showing Emoji, but you need to install that font
first.

bug#65993: 29.1; emoji-insert show nothing without font settings

[awrhygty]

Eli Zaretskii <eliz@gnu.org> writes:

>> And once emoji-insert is called before font setting,
>> emoji-insert does not show emoji items after they are made visible in
>> emacs buffer with font settings.
>
> This is not a bug: you need to clear the font cache to get Emacs to
> react to newly-installed fonts without restarting Emacs.  Like this:
>
>   M-: (clear-font-cache) RET

Evaluating (clear-font-cache) have no effect on emoji-insert.
Perhaps menu items are cached in another place.

>> Here is a sample setting to display emojis:
>>   (set-fontset-font nil '(#x1f000 . #x1faff) "Segoe UI Emoji")
>
> Segoe UI Emoji is not a free font, so we cannot advertise it in Emacs.
> You could tell Emacs about any fonts you have that support Emoji by
> using the function w32-find-non-USB-fonts, see its doc string for
> details.

Segoe UI was an only font for most of emojis installed in my PC.
In this case, I think emacs should find the font by default.

> In addition, Emacs 29 comes pre-configured to use the Noto Emoji font
> on MS-Windows for showing Emoji, but you need to install that font
> first.

Is Noto Color Emoji font is supported by emacs on MS-Windows?

bug#65993: 29.1; emoji-insert show nothing without font settings

[Eli Zaretskii]

> From: awrhygty@outlook.com
> Cc: 65993@debbugs.gnu.org
> Date: Sat, 16 Sep 2023 15:12:42 +0900
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> And once emoji-insert is called before font setting,
> >> emoji-insert does not show emoji items after they are made visible in
> >> emacs buffer with font settings.
> >
> > This is not a bug: you need to clear the font cache to get Emacs to
> > react to newly-installed fonts without restarting Emacs.  Like this:
> >
> >   M-: (clear-font-cache) RET
> 
> Evaluating (clear-font-cache) have no effect on emoji-insert.
> Perhaps menu items are cached in another place.

Maybe.  You can also try clear-composition-cache.  If that doesn't
help either, just restart the session.  Installing new fonts and
reconfiguring fontsets while running Emacs is not fully supported, due
to many caches we have for fonts and compositions.

> >> Here is a sample setting to display emojis:
> >>   (set-fontset-font nil '(#x1f000 . #x1faff) "Segoe UI Emoji")
> >
> > Segoe UI Emoji is not a free font, so we cannot advertise it in Emacs.
> > You could tell Emacs about any fonts you have that support Emoji by
> > using the function w32-find-non-USB-fonts, see its doc string for
> > details.
> 
> Segoe UI was an only font for most of emojis installed in my PC.
> In this case, I think emacs should find the font by default.

It cannot, not without help (or opening each and every font, which
would be prohibitively expensive), because the fonts don't announce
support for Emoji in the Unicode Subset Bits, which are used on
Windows to decide which fonts support a given script.  That's why
Emacs has the w32-find-non-USB-fonts function: it configures the font
search algorithm to consider fonts that support scripts, such as
Emoji, which don't have those bits allocated to them.  You need to run
that function just once, and as long as you don't install new fonts,
you can keep its result in your .emacs.

> > In addition, Emacs 29 comes pre-configured to use the Noto Emoji font
> > on MS-Windows for showing Emoji, but you need to install that font
> > first.
> 
> Is Noto Color Emoji font is supported by emacs on MS-Windows?

No.  But Noto Emoji, the black-and-white variant, is supported, and
Emacs will use it for Emoji if it's installed.

bug#65993: 29.1; emoji-insert show nothing without font settings

[awrhygty]

Eli Zaretskii <eliz@gnu.org> writes:

>> >> And once emoji-insert is called before font setting,
>> >> emoji-insert does not show emoji items after they are made visible in
>> >> emacs buffer with font settings.
>> >
>> > This is not a bug: you need to clear the font cache to get Emacs to
>> > react to newly-installed fonts without restarting Emacs.  Like this:
>> >
>> >   M-: (clear-font-cache) RET
>> 
>> Evaluating (clear-font-cache) have no effect on emoji-insert.
>> Perhaps menu items are cached in another place.
>
> Maybe.  You can also try clear-composition-cache.  If that doesn't
> help either, just restart the session.  Installing new fonts and
> reconfiguring fontsets while running Emacs is not fully supported, due
> to many caches we have for fonts and compositions.

The definition of emoji-insert is as below:
  (defun emoji-insert ()
    "Choose and insert an emoji glyph."
    (interactive "*")
    (emoji--init)
    (unless (fboundp 'emoji--command-Emoji)
      (emoji--define-transient))
    (funcall (intern "emoji--command-Emoji")))

The function emoji--init evaluates (require 'emoji-labels).
In the file emoji-labels.el, the variable emoji--labels is defined.
Its value is a hierarchical alist of emoji menu.
Then, inside the function emoji--init, non-displayable emojis in
emoji--labels are removed from alist.
After the function emoji--init, many functions are automatically
generated. Each of them corresponds to one item of emoji menu,
from which non-displayable emojis are already removed.
Finally emoji-insert calls the function of the top menu.

To change the emoji menu, I need to evaluate:
  (progn (load emoji-labels) (emoji--init))
This form resets emoji--labels and redefines menu functions.

I think the removal of non-displayable emojis is unnecessary.
If hex numbers are displayed, users can notice there are more emojis
than displayable. If it is difficult to find displayable emojis among
many non-displayable emojis, the menu classification is not
appropriate. Too many emojis in one menu node make user unable to find
the purpose emoji.

bug#65993: 29.1; emoji-insert show nothing without font settings

[Eli Zaretskii]

> From: awrhygty@outlook.com
> Cc: 65993@debbugs.gnu.org
> Date: Sun, 17 Sep 2023 18:37:02 +0900
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> >> And once emoji-insert is called before font setting,
> >> >> emoji-insert does not show emoji items after they are made visible in
> >> >> emacs buffer with font settings.
> >> >
> >> > This is not a bug: you need to clear the font cache to get Emacs to
> >> > react to newly-installed fonts without restarting Emacs.  Like this:
> >> >
> >> >   M-: (clear-font-cache) RET
> >> 
> >> Evaluating (clear-font-cache) have no effect on emoji-insert.
> >> Perhaps menu items are cached in another place.
> >
> > Maybe.  You can also try clear-composition-cache.  If that doesn't
> > help either, just restart the session.  Installing new fonts and
> > reconfiguring fontsets while running Emacs is not fully supported, due
> > to many caches we have for fonts and compositions.
> 
> The definition of emoji-insert is as below:
>   (defun emoji-insert ()
>     "Choose and insert an emoji glyph."
>     (interactive "*")
>     (emoji--init)
>     (unless (fboundp 'emoji--command-Emoji)
>       (emoji--define-transient))
>     (funcall (intern "emoji--command-Emoji")))
> 
> The function emoji--init evaluates (require 'emoji-labels).
> In the file emoji-labels.el, the variable emoji--labels is defined.
> Its value is a hierarchical alist of emoji menu.
> Then, inside the function emoji--init, non-displayable emojis in
> emoji--labels are removed from alist.
> After the function emoji--init, many functions are automatically
> generated. Each of them corresponds to one item of emoji menu,
> from which non-displayable emojis are already removed.
> Finally emoji-insert calls the function of the top menu.
> 
> To change the emoji menu, I need to evaluate:
>   (progn (load emoji-labels) (emoji--init))
> This form resets emoji--labels and redefines menu functions.
> 
> I think the removal of non-displayable emojis is unnecessary.
> If hex numbers are displayed, users can notice there are more emojis
> than displayable. If it is difficult to find displayable emojis among
> many non-displayable emojis, the menu classification is not
> appropriate. Too many emojis in one menu node make user unable to find
> the purpose emoji.

Lars, any comments?