bug#20727: 24.5; Font fallback doesn't work for the Emoji range

bug#20727: 24.5; Font fallback doesn't work for the Emoji range

[Vasilij Schneidermann]

[-- Attachment #1: Type: text/plain, Size: 3580 bytes --]

I did install the Symbola font to be able to display glyphs for code
points in the Emoji range which works for applications like my browser,
however Emacs isn't able of displaying these glyphs despite that
measure.  To reproduce the issue on a Linux system, it's sufficient to
have the Symbola font installed and with a default font different from
it to insert a character like PILE OF POO with C-x 8 RET which yields a
rectangle with numbers in it.  Temporarily switching the default font
to Symbola with M-x set-frame-font is one possible workaround to display
the glyph, another one is modifying the default fontset to explicitly
make use of that font for the Emoji range:

    (set-fontset-font "fontset-default" nil "Symbola"
                      (selected-frame) 'append)

I find this behaviour sort of puzzling as inspection of a buffer as
produced with M-x view-hello-file and M-x describe-char on the various
scripts confirms that Emacs managed choosing suitable fallback fonts for
scripts not covered by my default font.  Can someone enlighten me to
what exactly is going on here?



In GNU Emacs 24.5.1 (x86_64-unknown-linux-gnu, GTK+ Version 3.16.2)
 of 2015-04-20 on bitzer.hoetzel.info
Windowing system distributor `The X.Org Foundation', version 11.0.11701000
System Description: Arch Linux

Configured using:
 `configure --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib
 --localstatedir=/var --with-x-toolkit=gtk3 --with-xft
 'CFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong
 --param=ssp-buffer-size=4' CPPFLAGS=-D_FORTIFY_SOURCE=2
 LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro'

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.

Load-path shadows:
None found.

Features:
(shadow sort gnus-util mail-extr emacsbug message format-spec rfc822 mml
easymenu mml-sec mm-decode mm-bodies mm-encode mail-parse rfc2231
mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums
mm-util help-fns mail-prsvr mail-utils time-date tooltip electric
uniquify ediff-hook vc-hooks lisp-float-type mwheel x-win x-dnd tool-bar
dnd fontset image regexp-opt fringe tabulated-list newcomment lisp-mode
prog-mode register page menu-bar rfn-eshadow timer select scroll-bar
mouse jit-lock font-lock syntax facemenu font-core frame cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese hebrew greek romanian slovak czech european ethiopic indian
cyrillic chinese case-table epa-hook jka-cmpr-hook help simple abbrev
minibuffer nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote make-network-process
dbusbind gfilenotify dynamic-setting system-font-setting
font-render-setting move-toolbar gtk x-toolkit x multi-tty emacs)

Memory information:
((conses 16 70978 6876)
 (symbols 48 17559 0)
 (miscs 40 35 138)
 (strings 32 9044 4472)
 (string-bytes 1 248344)
 (vectors 16 8908)
 (vector-slots 8 383094 18351)
 (floats 8 63 213)
 (intervals 56 177 0)
 (buffers 960 11)
 (heap 1024 42438 899))

[-- Attachment #2: Type: text/html, Size: 4016 bytes --]

bug#20727: 24.5; Font fallback doesn't work for the Emoji range

[Eli Zaretskii]

> Date: Wed, 3 Jun 2015 19:22:42 +0200
> From: Vasilij Schneidermann <v.schneidermann@gmail.com>
> 
> I did install the Symbola font to be able to display glyphs for code
> points in the Emoji range which works for applications like my browser,
> however Emacs isn't able of displaying these glyphs despite that
> measure. To reproduce the issue on a Linux system, it's sufficient to
> have the Symbola font installed and with a default font different from
> it to insert a character like PILE OF POO with C-x 8 RET which yields a
> rectangle with numbers in it. Temporarily switching the default font
> to Symbola with M-x set-frame-font is one possible workaround to display
> the glyph, another one is modifying the default fontset to explicitly
> make use of that font for the Emoji range:
> 
> (set-fontset-font "fontset-default" nil "Symbola"
> (selected-frame) 'append)

This usually means you have some other font installed that claims to
cover the Emoji block, but doesn't actually cover this character.  Try
to find that font, and either uninstall it, or make it member of the
list in face-ignored-fonts, or customize your fontset like you show
above (there's nothing wrong with giving Emacs some help with the
fonts you have).

bug#20727: 24.5; Font fallback doesn't work for the Emoji range

[Vasilij Schneidermann]

[-- Attachment #1: Type: text/plain, Size: 732 bytes --]

> This usually means you have some other font installed that claims to
> cover the Emoji block, but doesn't actually cover this character.  Try
> to find that font, and either uninstall it, or make it member of the
> list in face-ignored-fonts, or customize your fontset like you show
> above (there's nothing wrong with giving Emacs some help with the
> fonts you have).

I did remove all user-installed fonts except Symbola to make sure it's just
the X and PS core fonts, yet the issue is still present. Therefore I can't
quite believe it to be linked to rogue fonts, another fact supporting my
hypothesis would be that I'm far from the only person experiencing this.
BTW, how does Emacs pick fonts for the scripts it encounters?

[-- Attachment #2: Type: text/html, Size: 832 bytes --]

bug#20727: 24.5; Font fallback doesn't work for the Emoji range

[Glenn Morris]

FWIW, I see the same as the OP on two different systems.

Eli Zaretskii wrote:

> This usually means you have some other font installed that claims to
> cover the Emoji block, but doesn't actually cover this character.

Is there a way to test that?

bug#20727: 24.5; Font fallback doesn't work for the Emoji range

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: Vasilij Schneidermann <v.schneidermann@gmail.com>,  20727@debbugs.gnu.org
> Date: Sun, 07 Jun 2015 14:09:27 -0400
> 
> FWIW, I see the same as the OP on two different systems.
> 
> Eli Zaretskii wrote:
> 
> > This usually means you have some other font installed that claims to
> > cover the Emoji block, but doesn't actually cover this character.
> 
> Is there a way to test that?

Try setting font-log to nil, and then look at its value after typing
the character.  The log is quite cryptic, but it does list the fonts.

bug#20727: 24.5; Font fallback doesn't work for the Emoji range

[Glenn Morris]

Using the unpleasant example character from the initial report, here are
what seem to be the relevant font-log entries:


(open "-unknown-DejaVu Sans Mono-normal-normal-normal-*-m-0-iso10646-1:script=symbol" "xft:-unknown-DejaVu Sans Mono-normal-normal-normal-*-13-*-m-0-iso10646-1")
 (sort-by "-*-normal-normal-normal-*-13-*" "xft:-unknown-DejaVu Sans Mono-normal-normal-normal-*-m-0-iso10646-1")
 (list "-unknown-DejaVu Sans Mono-*-i-isoso1010646-1:script=symbol"
       ["-unknown-DejaVu Sans Mono-normal-normal-normal-*-m-0-iso10646-1" "-unknown-DejaVu Sans Mono-bold-oblique-normal-*-m-0-isoso1010646-1" "-unknown-DejaVu Sans Mono-bold-normal-normal-*-m-0-isoso1010646-1" "-unknown-DejaVu Sans Mono-normal-oblique-normal-*-m-0-iso10646-1" "-unknown-DejaVu Sans Mono-normal-oblique-normal-*-m-0-iso10646-1" "-unknown-DejaVu Sans Mono-bold-normal-normal-*-m-0-isoso1010646-1" "-unknown-DejaVu Sans Mono-bold-oblique-normal-*-m-0-isoso1010646-1" "-unknown-DejaVu Sans Mono-normal-normal-normal-*-m-0-iso10646-1"])
 (xfont-list "-unknown-DejaVu Sans Mono-*-*-*-*-*-*-*-*-*-*-iso10646-1" nil)
 (ftfont-list "-unknown-DejaVu Sans Mono-*-iso10646-1:script=symbol"
	      ("-unknown-DejaVu Sans Mono-normal-normal-normal-*-m-0-iso10646-1" "-unknown-DejaVu Sans Mono-bold-oblique-normal-*-m-0-isoso1010646-1" "-unknown-DejaVu Sans Mono-bold-normal-normal-*-m-0-isoso1010646-1" "-unknown-DejaVu Sans Mono-normal-oblique-normal-*-m-0-iso10646-1" "-unknown-DejaVu Sans Mono-normal-oblique-normal-*-m-0-iso10646-1" "-unknown-DejaVu Sans Mono-bold-normal-normal-*-m-0-isoso1010646-1" "-unknown-DejaVu Sans Mono-bold-oblique-normal-*-m-0-isoso1010646-1" "-unknown-DejaVu Sans Mono-normal-normal-normal-*-m-0-iso10646-1"))
 (default\ fontset:\ font\ for 128169 nil)

 

bug#20727: 24.5; Font fallback doesn't work for the Emoji range

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: 20727@debbugs.gnu.org,  v.schneidermann@gmail.com
> Date: Sun, 07 Jun 2015 20:15:55 -0400
> 
> Using the unpleasant example character from the initial report, here are
> what seem to be the relevant font-log entries:
> 
> 
> (open "-unknown-DejaVu Sans Mono-normal-normal-normal-*-m-0-iso10646-1:script=symbol" "xft:-unknown-DejaVu Sans Mono-normal-normal-normal-*-13-*-m-0-iso10646-1")

The "open" entry means Emacs decided to use that font.  Does "DejaVu
Sans Mono" have glyphs for those characters?

bug#20727: 24.5; Font fallback doesn't work for the Emoji range

[Vasilij Schneidermann]

> The "open" entry means Emacs decided to use that font.  Does "DejaVu
> Sans Mono" have glyphs for those characters?

I use `gucharmap` to look up glyphs in fonts, you can find it with C-f
by its name.  Selecting "DejaVu Sans Mono" as font and the "Show only
glyphs from this font" option from the "View" menu confirms that this
font doesn't have a glyph for it.

What struck me as odd however is that Emacs appears to select glyphs by
script and that there's no "Symbol" script in this font viewer (or more
likely, not in Unicode at all).  Perhaps here lies the problem.

bug#20727: 24.5; Font fallback doesn't work for the Emoji range

[Eli Zaretskii]

> Date: Mon, 8 Jun 2015 07:43:04 +0200
> From: Vasilij Schneidermann <v.schneidermann@gmail.com>
> Cc: Glenn Morris <rgm@gnu.org>, 20727@debbugs.gnu.org
> 
> > The "open" entry means Emacs decided to use that font.  Does "DejaVu
> > Sans Mono" have glyphs for those characters?
> 
> I use `gucharmap` to look up glyphs in fonts, you can find it with C-f
> by its name.  Selecting "DejaVu Sans Mono" as font and the "Show only
> glyphs from this font" option from the "View" menu confirms that this
> font doesn't have a glyph for it.

Does the full value of font-log suggest that Emacs even tried to
consider the Symbola font at some point?  My reading of the code seems
to indicate that Emacs won't try any font that's not in the fontset,
for characters that don't have a charset ID (which is what happens
with symbols, see below).

> What struck me as odd however is that Emacs appears to select glyphs by
> script and that there's no "Symbol" script in this font viewer (or more
> likely, not in Unicode at all).  Perhaps here lies the problem.

There's elaborate setup in fontset.el for selecting fonts by script,
so the fact that HELLO displays correctly is not a miracle.  Moreover,
for characters that belong to known charsets -- something that happens
for every letter in some language or script -- Emacs queries available
fonts by charset, so it can find fonts that are not in the fontset, I
think.

However 'symbol' is not a script nor a charset; Emacs conflates into
that pseudo-script all the Unicode blocks that define symbols of some
kind, for which the OTF spec does not define a script tag (for the
list of tags see
https://www.microsoft.com/typography/otspec/scripttags.htm).  We can
also look up fonts by family, registry, and charset, but none of this
helps with symbols, like it does with "real" scripts and character
sets.  So I think we pretty much depend on the fonts in the default
fontset for supporting symbols in general and Emoji in particular.

Therefore, I think we should augment the default fontset to include
good fonts for symbols for which the "usual" fonts rarely if ever
contain glyphs.

The problem is that most such fonts are not free.  Is Symbola free?

There's GNU Unifont, which is free, and is already in the default
fontset, but IMO many of its glyphs are really of low quality.  Do you
have that installed?  I'm guessing not.

bug#20727: 24.5; Font fallback doesn't work for the Emoji range

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

> The problem is that most such fonts are not free.  Is Symbola free?

http://users.teilar.gr/~g1951d/
"free for any use"

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

bug#20727: 24.5; Font fallback doesn't work for the Emoji range

[Vasilij Schneidermann]

> The problem is that most such fonts are not free.  Is Symbola free?

Symbola comes with a custom license, but is packaged with a GPL license
on Debian in the ttf-ancient-fonts package[1].  Another alternative font
would be Quivira[2] which spans a comparable amount of characters and is
distributed under a custom license as well (which could be probably
packaged in the same manner as Symbola).

[1]: http://metadata.ftp-master.debian.org/changelogs//main/t/ttf-ancient-fonts/ttf-ancient-fonts_2.57-1_copyright
[2]: http://www.quivira-font.com/

bug#20727: 24.5; Font fallback doesn't work for the Emoji range

[Eli Zaretskii]

> From: Andreas Schwab <schwab@suse.de>
> Cc: Vasilij Schneidermann <v.schneidermann@gmail.com>,  20727@debbugs.gnu.org
> Date: Mon, 08 Jun 2015 16:52:48 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > The problem is that most such fonts are not free.  Is Symbola free?
> 
> http://users.teilar.gr/~g1951d/
> "free for any use"

Thanks, I pushed a change that I hope will improve the default
fontset's coverage of the various symbol characters.

bug#20727: 24.5; Font fallback doesn't work for the Emoji range

[Andy Moreton]

On Mon 08 Jun 2015, Eli Zaretskii wrote:

>> From: Andreas Schwab <schwab@suse.de>
>> Cc: Vasilij Schneidermann <v.schneidermann@gmail.com>,  20727@debbugs.gnu.org
>> Date: Mon, 08 Jun 2015 16:52:48 +0200
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > The problem is that most such fonts are not free.  Is Symbola free?
>> 
>> http://users.teilar.gr/~g1951d/
>> "free for any use"
>
> Thanks, I pushed a change that I hope will improve the default
> fontset's coverage of the various symbol characters.

This has caused a regression for me on the mingw64 build of emacs.

I use various box drawing characters in gnus to show threading, and
these normally come from DejaVu Mono (the default font).

After this change, Emacs uses Batang Che for the box drawing characters,
unless I add code to reverse the effects of the patch.

Can you please improve the patch to avoid overriding the default font
with fonts that don't exist, or at least make it easier for the user to
choose the fonts chosen for the symbol-subgroup and box drawing
characters without copying the character ranges.


    AndyM

bug#20727: 24.5; Font fallback doesn't work for the Emoji range

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Tue, 09 Jun 2015 12:48:16 +0100
> 
> On Mon 08 Jun 2015, Eli Zaretskii wrote:
> 
> >> From: Andreas Schwab <schwab@suse.de>
> >> Cc: Vasilij Schneidermann <v.schneidermann@gmail.com>,  20727@debbugs.gnu.org
> >> Date: Mon, 08 Jun 2015 16:52:48 +0200
> >> 
> >> Eli Zaretskii <eliz@gnu.org> writes:
> >> 
> >> > The problem is that most such fonts are not free.  Is Symbola free?
> >> 
> >> http://users.teilar.gr/~g1951d/
> >> "free for any use"
> >
> > Thanks, I pushed a change that I hope will improve the default
> > fontset's coverage of the various symbol characters.
> 
> This has caused a regression for me on the mingw64 build of emacs.
> 
> I use various box drawing characters in gnus to show threading, and
> these normally come from DejaVu Mono (the default font).
> 
> After this change, Emacs uses Batang Che for the box drawing characters,
> unless I add code to reverse the effects of the patch.

Not sure how that happened: AFAIK, my change mentioned neither Batang
Che nor DejaVu Mono.  Do you have in your customizations some setup
for fontsets that cover these characters?  If so, could you please
show these customizations?

> Can you please improve the patch to avoid overriding the default font
> with fonts that don't exist

Maybe I'm missing something, but my patch didn't override the default
font in any way that I'm aware of.  In fact, Emacs doesn't (well,
didn't until now, see below) consult the default font when it needs to
display characters such as box drawing; it always searches for the
proper font regardless of the coverage of the default font.  At least
that's AFAIU the code: this area in Emacs is notoriously
under-documented.

Anyway, I found and fixed an unintended consequence of my change,
please try the latest master.  Now, if the font used for ASCII face,
which AFAIU is the same as the frame's default font, has a glyph for
punctuation and other symbol characters, Emacs will use the default
font, instead of searching for another font.

> or at least make it easier for the user to choose the fonts chosen
> for the symbol-subgroup and box drawing characters without copying
> the character ranges.

Not sure what that means; how did you choose your fonts until now?  I
think this again hints on some font-related customizations you did, in
which case please show them.  My changes targeted the users who don't
do any fontset customizations and don't know how to do that; the
assumption was that whoever does customize their fontsets can easily
adapt.  Is that wrong?

bug#20727: 24.5; Font fallback doesn't work for the Emoji range

[Andy Moreton]

On Tue 09 Jun 2015, Eli Zaretskii wrote:

>> From: Andy Moreton <andrewjmoreton@gmail.com>
>> Date: Tue, 09 Jun 2015 12:48:16 +0100
>> 
>> On Mon 08 Jun 2015, Eli Zaretskii wrote:
>> 
>> >> From: Andreas Schwab <schwab@suse.de>
>> >> Cc: Vasilij Schneidermann <v.schneidermann@gmail.com>,  20727@debbugs.gnu.org
>> >> Date: Mon, 08 Jun 2015 16:52:48 +0200
>> >> 
>> >> Eli Zaretskii <eliz@gnu.org> writes:
>> >> 
>> >> > The problem is that most such fonts are not free.  Is Symbola free?
>> >> 
>> >> http://users.teilar.gr/~g1951d/
>> >> "free for any use"
>> >
>> > Thanks, I pushed a change that I hope will improve the default
>> > fontset's coverage of the various symbol characters.
>> 
>> This has caused a regression for me on the mingw64 build of emacs.
>> 
>> I use various box drawing characters in gnus to show threading, and
>> these normally come from DejaVu Mono (the default font).
>> 
>> After this change, Emacs uses Batang Che for the box drawing characters,
>> unless I add code to reverse the effects of the patch.
>
> Not sure how that happened: AFAIK, my change mentioned neither Batang
> Che nor DejaVu Mono.  Do you have in your customizations some setup
> for fontsets that cover these characters?  If so, could you please
> show these customizations?

All I have is to choose DejaVu Mono as the default font. Your changes
made the defqault fontset choose Symbola and FreeMono for symbols and
box drawing characters, but my system does not have thiose fonts, so it
chose something else instead.

>> Can you please improve the patch to avoid overriding the default font
>> with fonts that don't exist
>
> Maybe I'm missing something, but my patch didn't override the default
> font in any way that I'm aware of.  In fact, Emacs doesn't (well,
> didn't until now, see below) consult the default font when it needs to
> display characters such as box drawing; it always searches for the
> proper font regardless of the coverage of the default font.  At least
> that's AFAIU the code: this area in Emacs is notoriously
> under-documented.
>
> Anyway, I found and fixed an unintended consequence of my change,
> please try the latest master.  Now, if the font used for ASCII face,
> which AFAIU is the same as the frame's default font, has a glyph for
> punctuation and other symbol characters, Emacs will use the default
> font, instead of searching for another font.

Thanks Eli, this changes restores the original behaviour.

>> or at least make it easier for the user to choose the fonts chosen
>> for the symbol-subgroup and box drawing characters without copying
>> the character ranges.
>
> Not sure what that means; how did you choose your fonts until now?  I
> think this again hints on some font-related customizations you did, in
> which case please show them.  My changes targeted the users who don't
> do any fontset customizations and don't know how to do that; the
> assumption was that whoever does customize their fontsets can easily
> adapt.  Is that wrong?

The only font-related customisation I have is:

  (set-face-attribute 'default nil        :font "DejaVu Sans Mono-9")
  (set-face-attribute 'variable-pitch nil :font "Arial-10")

    (setq default-frame-alist
          (append `((background-color . "gray85")
                    (font             .  "DejaVu Sans Mono-9")
                    ))
                  default-frame-alist))

I don't have any fontset customisation at all, but the earlier change
still made emacs choose a different font for symbols.

    AndyM

bug#20727: 24.5; Font fallback doesn't work for the Emoji range

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Tue, 09 Jun 2015 17:29:30 +0100
> 
> >> After this change, Emacs uses Batang Che for the box drawing characters,
> >> unless I add code to reverse the effects of the patch.
> >
> > Not sure how that happened: AFAIK, my change mentioned neither Batang
> > Che nor DejaVu Mono.  Do you have in your customizations some setup
> > for fontsets that cover these characters?  If so, could you please
> > show these customizations?
> 
> All I have is to choose DejaVu Mono as the default font. Your changes
> made the defqault fontset choose Symbola and FreeMono for symbols and
> box drawing characters, but my system does not have thiose fonts, so it
> chose something else instead.

What I cannot understand is why it chose Batang Che, when it chose
DejaVu Mono before.  Before my changes, it should have examined Batang
Che before DejaVu Mono, so if it was "good" after my changes, it
should have been good before them.  AFAICS, fonts are generally
examined in the same order.

Probably some additional surprise lurking in the font-selection code.

> > Anyway, I found and fixed an unintended consequence of my change,
> > please try the latest master.  Now, if the font used for ASCII face,
> > which AFAIU is the same as the frame's default font, has a glyph for
> > punctuation and other symbol characters, Emacs will use the default
> > font, instead of searching for another font.
> 
> Thanks Eli, this changes restores the original behaviour.

It actually introduces behavior that was never there to begin with
(except as a not-entirely-correct variation in the NS build).  But at
least now I can explain what Emacs does, and why ;-)

> The only font-related customisation I have is:
> 
>   (set-face-attribute 'default nil        :font "DejaVu Sans Mono-9")
>   (set-face-attribute 'variable-pitch nil :font "Arial-10")
> 
>     (setq default-frame-alist
>           (append `((background-color . "gray85")
>                     (font             .  "DejaVu Sans Mono-9")
>                     ))
>                   default-frame-alist))
> 
> I don't have any fontset customisation at all, but the earlier change
> still made emacs choose a different font for symbols.

Strange.

bug#20727: 24.5; Font fallback doesn't work for the Emoji range

[Glenn Morris]

The original issue is now fixed for me, thanks.

bug#20727: 24.5; Font fallback doesn't work for the Emoji range

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Cc: Andy Moreton <andrewjmoreton@gmail.com>,  20727@debbugs.gnu.org
> Date: Fri, 12 Jun 2015 12:05:18 -0400
> 
> 
> The original issue is now fixed for me, thanks.

Thanks, closing.

bug#20727: 24.5; Font fallback doesn't work for the Emoji range

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 1274 bytes --]

The recent change to fontsel.el messed up my display when I was trying 
to test obsolescent 8-bit environments.  I can reproduce the problem on 
Fedora 21 x86-64 by running:

src/emacs -Q -font 
-misc-fixed-medium-r-semicondensed--0-0-75-75-c-0-iso8859-1

and then by typing:

abc C-x 8 [ def C-x 8 ] ghi

This inserts "abc‘def’ghi" into *scratch*.  The display looks like "abc  
‘def’  ghi" with huge spaces around the quotes (see attached 
screenshot).  There are similar problems with many other characters.  
Apparently this is because I don't have the Symbola font installed, so 
the change caused Emacs to fall back on Chinese double-width quotes 
rather than on the font it used before (which is -misc-liberation 
serif-medium-r-normal--13-94-100-100-p-74-iso10646-1 for the above 
invocation of Emacs).

I imagine that there will be similar problems with choosing Symbola even 
if it's available, as users may prefer the font they've selected 
explicitly, so perhaps the change should be enabled only for characters 
that are not already available in the user-selected font?  In the 
meantime, in master commit 203e84c6cf9b8356e376cc748b5ed331df96dc9e I 
worked around my problem by skipping the change if Symbola is not installed.

[-- Attachment #2: Screenshot from 2015-06-12 12:59:19.png --]
[-- Type: image/png, Size: 1509 bytes --]

bug#20727: 24.5; Font fallback doesn't work for the Emoji range

[Eli Zaretskii]

> Date: Fri, 12 Jun 2015 13:57:26 -0700
> From: Paul Eggert <eggert@cs.ucla.edu>
> CC: Eli Zaretskii <eliz@gnu.org>, 
>  Vasilij Schneidermann <v.schneidermann@gmail.com>,
>  Andy Moreton <andrewjmoreton@gmail.com>
> 
> The recent change to fontsel.el messed up my display when I was trying 
> to test obsolescent 8-bit environments.  I can reproduce the problem on 
> Fedora 21 x86-64 by running:
> 
> src/emacs -Q -font 
> -misc-fixed-medium-r-semicondensed--0-0-75-75-c-0-iso8859-1
> 
> and then by typing:
> 
> abc C-x 8 [ def C-x 8 ] ghi
> 
> This inserts "abc‘def’ghi" into *scratch*.  The display looks like "abc  
> ‘def’  ghi" with huge spaces around the quotes (see attached 
> screenshot).  There are similar problems with many other characters.  
> Apparently this is because I don't have the Symbola font installed, so 
> the change caused Emacs to fall back on Chinese double-width quotes 
> rather than on the font it used before (which is -misc-liberation 
> serif-medium-r-normal--13-94-100-100-p-74-iso10646-1 for the above 
> invocation of Emacs).

I recommend to either install Symbola (preferred) or customize the
default fontset on your system to use the font it used previously, if
you like it better than Symbola.

> I imagine that there will be similar problems with choosing Symbola even 
> if it's available, as users may prefer the font they've selected 
> explicitly

With the current master (even before your change), Symbola is
preferred only for punctuation and other symbols, and only if the
default font doesn't support those characters.  I don't expect users
to prefer some specific font for those, except their default font.
User preferences are expected in Unicode ranges that belong to users'
scripts, not for symbols and punctuation that their default font
doesn't already support.  In the rare cases where users do prefer
specific fonts for those characters, they can customize the fontsets.

> so perhaps the change should be enabled only for characters 
> that are not already available in the user-selected font?

That's what the code already does.  Are you saying that your default
font supported those quotes, and yet Emacs didn't use that font for
them?  You say above "fall back", and mention a non-default font,
which I understand to mean your default font didn't support those
characters.

bug#20727: 24.5; Font fallback doesn't work for the Emoji range

[Eli Zaretskii]

> Date: Fri, 12 Jun 2015 13:57:26 -0700
> From: Paul Eggert <eggert@cs.ucla.edu>
> CC: Eli Zaretskii <eliz@gnu.org>, 
>  Vasilij Schneidermann <v.schneidermann@gmail.com>,
>  Andy Moreton <andrewjmoreton@gmail.com>
> 
> In the meantime, in master commit
> 203e84c6cf9b8356e376cc748b5ed331df96dc9e I worked around my problem
> by skipping the change if Symbola is not installed.

I had to revert that commit, sorry.  I do have Symbola installed, and
yet that change caused the default-fontset not to include the
customization for symbols and punctuation.  I guess the machinery for
looking up fonts is not yet fully set up when setup-default-fontset is
called during initialization, and find-font always fails.

So if the problem you bumped into is indeed a serious one, and telling
users to install Symbola or customize their fontsets is not good
enough, we will have to look for another solution for this issue.

I'll try to figure out how come a non-existing font in the default
fontset suddenly changes which fallback font is selected, that's
something I didn't expect to happen.

bug#20727: 24.5; Font fallback doesn't work for the Emoji range

[Eli Zaretskii]

> Date: Fri, 12 Jun 2015 13:57:26 -0700
> From: Paul Eggert <eggert@cs.ucla.edu>
> CC: Eli Zaretskii <eliz@gnu.org>, 
>  Vasilij Schneidermann <v.schneidermann@gmail.com>,
>  Andy Moreton <andrewjmoreton@gmail.com>
> 
> src/emacs -Q -font 
> -misc-fixed-medium-r-semicondensed--0-0-75-75-c-0-iso8859-1
> 
> and then by typing:
> 
> abc C-x 8 [ def C-x 8 ] ghi
> 
> This inserts "abc‘def’ghi" into *scratch*.  The display looks like "abc  
> ‘def’  ghi" with huge spaces around the quotes (see attached 
> screenshot).  There are similar problems with many other characters.  
> Apparently this is because I don't have the Symbola font installed, so 
> the change caused Emacs to fall back on Chinese double-width quotes 
> rather than on the font it used before (which is -misc-liberation 
> serif-medium-r-normal--13-94-100-100-p-74-iso10646-1 for the above 
> invocation of Emacs).

Please tell the exact spec of the font with Chinese double-width
quotes Emacs selected on that system (like you show above for the
Liberation Serif font).  Also, please send (as an attachment) the
entire contents of the *Help* buffer produced by "M-x describe-fontset
RET RET" on that system.  I'd like to see what can be done to avoid
that, even if Symbola is not installed.

Thanks.

bug#20727: 24.5; Font fallback doesn't work for the Emoji range

[Eli Zaretskii]

> Date: Sat, 13 Jun 2015 12:12:48 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: v.schneidermann@gmail.com, andrewjmoreton@gmail.com, 20727@debbugs.gnu.org
> 
> > Date: Fri, 12 Jun 2015 13:57:26 -0700
> > From: Paul Eggert <eggert@cs.ucla.edu>
> > CC: Eli Zaretskii <eliz@gnu.org>, 
> >  Vasilij Schneidermann <v.schneidermann@gmail.com>,
> >  Andy Moreton <andrewjmoreton@gmail.com>
> > 
> > src/emacs -Q -font 
> > -misc-fixed-medium-r-semicondensed--0-0-75-75-c-0-iso8859-1
> > 
> > and then by typing:
> > 
> > abc C-x 8 [ def C-x 8 ] ghi
> > 
> > This inserts "abc‘def’ghi" into *scratch*.  The display looks like "abc  
> > ‘def’  ghi" with huge spaces around the quotes (see attached 
> > screenshot).  There are similar problems with many other characters.  
> > Apparently this is because I don't have the Symbola font installed, so 
> > the change caused Emacs to fall back on Chinese double-width quotes 
> > rather than on the font it used before (which is -misc-liberation 
> > serif-medium-r-normal--13-94-100-100-p-74-iso10646-1 for the above 
> > invocation of Emacs).
> 
> Please tell the exact spec of the font with Chinese double-width
> quotes Emacs selected on that system (like you show above for the
> Liberation Serif font).  Also, please send (as an attachment) the
> entire contents of the *Help* buffer produced by "M-x describe-fontset
> RET RET" on that system.  I'd like to see what can be done to avoid
> that, even if Symbola is not installed.

Please also see if the latest master fixes the problem for you; if it
does, then you don't need to send all of the information I requested.
(I think I figured out why the original change caused Emacs to choose
a different font than what it would chose before that change.)

bug#20727: 24.5; Font fallback doesn't work for the Emoji range

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 1322 bytes --]

Eli Zaretskii wrote:
> Please also see if the latest master fixes the problem for you; if it
> does, then you don't need to send all of the information I requested.

I can't easily try that until Monday, sorry.  But I now have a different problem 
with the patch.

I'm on a machine with Symbola installed now, and with the latest master version 
(commit eb92f89c2125aaf8fdf93cdd85ab46ae278dd950) the display is way worse than 
it was before.  See attached screenshots of Emacs 24.4 and latest master 
displaying the following text in a fundamental-mode buffer:

  abc‘def’ghi
  abc“def”ghi
  abc≤def

I ran Emacs with the arguments "-Q -r -font 
-misc-fixed-medium-r-semicondensed--13-120-75-75-c-60-iso8859-1".  Emacs 
previously substituted 
-misc-fixed-medium-r-semicondensed--13-120-75-75-c-60-iso10646-1 for the 
non-alphabetic characters, and this worked well: it's the same font, really, and 
the substitute characters are legible and match the default font well.  In 
contrast, Symbola is varying width, the characters don't match the default font, 
and the characters are in some cases nearly illegible.

Why is Emacs using Symbola in a setup that has good Unicode characters already? 
  Isn't the idea to use Symbola only as a fallback, when the existing fonts 
don't work?

[-- Attachment #2: Screenshot from 2015-06-13 08:49:51.png --]
[-- Type: image/png, Size: 501 bytes --]

[-- Attachment #3: Screenshot from 2015-06-13 08:50:20.png --]
[-- Type: image/png, Size: 1148 bytes --]

bug#20727: 24.5; Font fallback doesn't work for the Emoji range

[Eli Zaretskii]

> Date: Sat, 13 Jun 2015 09:01:38 -0700
> From: Paul Eggert <eggert@cs.ucla.edu>
> CC: v.schneidermann@gmail.com, andrewjmoreton@gmail.com, 
>  20727@debbugs.gnu.org
> 
> I'm on a machine with Symbola installed now, and with the latest master version 
> (commit eb92f89c2125aaf8fdf93cdd85ab46ae278dd950) the display is way worse than 
> it was before.  See attached screenshots of Emacs 24.4 and latest master 
> displaying the following text in a fundamental-mode buffer:
> 
>   abc‘def’ghi
>   abc“def”ghi
>   abc≤def

I don't really see why it's "way worth".  I see quite similar
displays.

> I ran Emacs with the arguments "-Q -r -font 
> -misc-fixed-medium-r-semicondensed--13-120-75-75-c-60-iso8859-1".  Emacs 
> previously substituted 
> -misc-fixed-medium-r-semicondensed--13-120-75-75-c-60-iso10646-1 for the 
> non-alphabetic characters, and this worked well: it's the same font, really, and 
> the substitute characters are legible and match the default font well.  In 
> contrast, Symbola is varying width, the characters don't match the default font, 
> and the characters are in some cases nearly illegible.

The Symbola font looks much more crisp on my system, FWIW.  Which font
back-end did you configure Emacs to use?

Also, if you use the iso10646-1 variant instead of the iso8859-1 as
the default font, doesn't that fix the problem, in that the default
font is used for both ASCII and the punctuation characters?  That's
what happens on my system: the default font, which is Courier New,
does included these quotes, so it is used when these characters need
to be displayed, and Emacs falls back on Symbola only for the more
rare symbols.

> Why is Emacs using Symbola in a setup that has good Unicode characters already? 

How would Emacs know where to find "good Unicode characters", in a
font with coverage that is good enough to show almost any character
that's there?  And how can Emacs know which fonts you like and don't
like?

>   Isn't the idea to use Symbola only as a fallback, when the existing fonts 
> don't work?

No, it's the other way around.  Experience shows that the existing
fonts are frequently inadequate, in that they claim support for
Unicode ranges where they actually support only a handful of glyphs.
Users then complain that they have decent fonts (like Symbola)
installed, but Emacs still shows some characters as boxes with hex
code, instead of using Symbola.

I tried to improve on that in my latest changes.  I do expect some
rough edges, since this area in Emacs is notoriously under-documented,
and we no longer have experts on board who are active enough to take
care of this.  So perhaps I made some mistakes with my changes, or
didn't use the right options.  But I don't think it's a good idea to
go back to the previous arrangement where any font that claimed
iso10646-1 support would be considered as covering symbols and
punctuation well, because that means restoring the problems I tried to
fix in the first place.

How about the following compromise: we exclude the ranges for the most
popular symbols and punctuation, which are reasonably well covered by
many fonts, from the characters for which we specify Symbola?  This
would include the quotes you've shown and a few other similar
characters.  (It would require some footwork for finding out which
characters are covered well by many fonts.)  Would that be good
enough?

bug#20727: 24.5; Font fallback doesn't work for the Emoji range

[Eli Zaretskii]

> Date: Sat, 13 Jun 2015 19:32:45 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: v.schneidermann@gmail.com, andrewjmoreton@gmail.com, 20727@debbugs.gnu.org
> 
> How about the following compromise: we exclude the ranges for the most
> popular symbols and punctuation, which are reasonably well covered by
> many fonts, from the characters for which we specify Symbola?  This
> would include the quotes you've shown and a few other similar
> characters.

Actually, an even better solution might be this: we explicitly specify
fixed-medium font with iso10646-1 registry for these popular
characters _before_ Symbola.  This is justified, I think, since the
standard-fontset-spec on X specifies a fixed-medium font, so fit is
iso10646-1 variant supports these commonly-used punctuation
characters, it is a better candidate than Symbola, while the latter
will still be used as fallback.

WDYT?  Could you try this on your system and see if it work?

bug#20727: 24.5; Font fallback doesn't work for the Emoji range

[Paul Eggert]

Eli Zaretskii wrote:

> I don't really see why it's "way worth".

Sorry, I mistyped (and used my Sylvester voice 
<https://en.wikipedia.org/wiki/Sylvester_%28Looney_Tunes%29>).  I meant that 
Symbola looks "way worse".  I assume that you can see that in the screenshots, 
it's just that you can't reproduce the problem on your machine.

> The Symbola font looks much more crisp on my system, FWIW.  Which font
> back-end did you configure Emacs to use?

Sorry, I don't know what "font back-end" means.  I'm running on Ubuntu 15.04 and 
configured Emacs with --enable-gcc-warnings.  'configure' outputs:

Configured for 'x86_64-unknown-linux-gnu'.

   Where should the build process find the source code?    .
   What compiler should emacs be built with?               gcc -std=gnu99 -g3 -O2
   Should Emacs use the GNU version of malloc?             yes
       (Using Doug Lea's new malloc from the GNU C Library.)
   Should Emacs use a relocating allocator for buffers?    no
   Should Emacs use mmap(2) for buffer allocation?         no
   What window system should Emacs use?                    x11
   What toolkit should Emacs use?                          GTK3
   Where do we find X Windows header files?                Standard dirs
   Where do we find X Windows libraries?                   Standard dirs
   Does Emacs use -lXaw3d?                                 no
   Does Emacs use -lXpm?                                   yes
   Does Emacs use -ljpeg?                                  yes
   Does Emacs use -ltiff?                                  yes
   Does Emacs use a gif library?                           yes -lgif
   Does Emacs use a png library?                           yes -lpng12
   Does Emacs use -lrsvg-2?                                yes
   Does Emacs use cairo?                                   no
   Does Emacs use imagemagick?                             yes
   Does Emacs support sound?                               yes
   Does Emacs use -lgpm?                                   yes
   Does Emacs use -ldbus?                                  yes
   Does Emacs use -lgconf?                                 yes
   Does Emacs use GSettings?                               yes
   Does Emacs use a file notification library?             yes -lgio (gfile)
   Does Emacs use access control lists?                    no
   Does Emacs use -lselinux?                               no
   Does Emacs use -lgnutls?                                yes
   Does Emacs use -lxml2?                                  yes
   Does Emacs use -lfreetype?                              yes
   Does Emacs use -lm17n-flt?                              yes
   Does Emacs use -lotf?                                   yes
   Does Emacs use -lxft?                                   yes
   Does Emacs directly use zlib?                           yes
   Does Emacs use toolkit scroll bars?                     yes

> Also, if you use the iso10646-1 variant instead of the iso8859-1 as
> the default font, doesn't that fix the problem

Yes.  The problem, I imagine, is that users will have put a fixed-width font of 
their own preference into their .Xdefaults or .Xresources file (e.g., 
"Emacs.font fixed").  This is reasonably common, and the recent change makes 
symbols look much worse, at least in my environment.

By the way, I can reproduce a similar problem with "emacs -Q -font fixed".  It's 
not as unreadable there, but it's still pretty bad: Symbola characters have a 
different width than the fixed-width characters that Emacs previously used, so 
source code listings no longer line up.

> existing
> fonts are frequently inadequate, in that they claim support for
> Unicode ranges where they actually support only a handful of glyphs.
> Users then complain that they have decent fonts (like Symbola)
> installed, but Emacs still shows some characters as boxes with hex
> code, instead of using Symbola.

This doesn't seem to be a problem in Ubuntu and/or Fedora (the systems I 
typically use), and on these environments the cure seems to be worse than the 
disease.  Is there some way we can heuristically tell whether we're in an 
environment where glyphs are often not supported?  For example, can we convert a 
sample glyph or two to a bitmap and see whether it looks like a boxed hex code? 
  (Just thinking out loud.)

> I don't think it's a good idea to
> go back to the previous arrangement where any font that claimed
> iso10646-1 support would be considered as covering symbols and
> punctuation well, because that means restoring the problems I tried to
> fix in the first place.

In that case I don't understand why emacs -Q -font 
-misc-fixed-medium-r-semicondensed--13-120-75-75-c-60-iso10646-1 works now.  If 
Emacs is supposed to prefer Symbola to other fonts when displaying symbols, why 
isn't it using Symbola in this case?

bug#20727: 24.5; Font fallback doesn't work for the Emoji range

[Paul Eggert]

Eli Zaretskii wrote:
> Actually, an even better solution might be this: we explicitly specify
> fixed-medium font with iso10646-1 registry for these popular
> characters_before_  Symbola.  This is justified, I think, since the
> standard-fontset-spec on X specifies a fixed-medium font, so fit is
> iso10646-1 variant supports these commonly-used punctuation
> characters, it is a better candidate than Symbola, while the latter
> will still be used as fallback.
>
> WDYT?  Could you try this on your system and see if it work?

I'd be happy to try, but I'm afraid this is an area I don't know well and I 
don't see why the existing code doesn't already do what you'r suggesting. 
setup-default-fontset invokes (set-fontset-font "fontset-default" ... "Symbola" 
nil 'prepend), and later invokes (set-fontset-font "fontset-default" nil '(nil . 
"iso10646-1") nil 'prepend), so doesn't that mean that the iso10646-1 fonts are 
prepended before Symbola and so should take priority over Symbola already?

Anyway, If you can send me a patch along the lines you're thinking, I can try it.

bug#20727: 24.5; Font fallback doesn't work for the Emoji range

[Eli Zaretskii]

> Date: Sat, 13 Jun 2015 10:07:09 -0700
> From: Paul Eggert <eggert@cs.ucla.edu>
> CC: v.schneidermann@gmail.com, andrewjmoreton@gmail.com, 
>  20727@debbugs.gnu.org
> 
> > The Symbola font looks much more crisp on my system, FWIW.  Which font
> > back-end did you configure Emacs to use?
> 
> Sorry, I don't know what "font back-end" means.

The answer is here:

>    Does Emacs use -lm17n-flt?                              yes
>    Does Emacs use -lotf?                                   yes
>    Does Emacs use -lxft?                                   yes

This means your font back-end is xft with libotf.  That's the most
advanced one we have on GNU and Unix systems, I think.  I thought
maybe you had something less advanced for some reason.

> > Also, if you use the iso10646-1 variant instead of the iso8859-1 as
> > the default font, doesn't that fix the problem
> 
> Yes. The problem, I imagine, is that users will have put a fixed-width font of 
> their own preference into their .Xdefaults or .Xresources file (e.g., 
> "Emacs.font fixed").  This is reasonably common, and the recent change makes 
> symbols look much worse, at least in my environment.

The question is how to avoid that without re-introducing the original
problems.

> By the way, I can reproduce a similar problem with "emacs -Q -font fixed".  It's 
> not as unreadable there, but it's still pretty bad: Symbola characters have a 
> different width than the fixed-width characters that Emacs previously used, so 
> source code listings no longer line up.

At some point, with sufficiently rare characters that are not well
covered by fixed-width fonts, this will always happen.  I don't think
it's a catastrophe; it is certainly better than having the boxes with
hex codes instead (which also make text misalign, btw).

> > existing
> > fonts are frequently inadequate, in that they claim support for
> > Unicode ranges where they actually support only a handful of glyphs.
> > Users then complain that they have decent fonts (like Symbola)
> > installed, but Emacs still shows some characters as boxes with hex
> > code, instead of using Symbola.
> 
> This doesn't seem to be a problem in Ubuntu and/or Fedora (the systems I 
> typically use)

I think you are judging by characters that are sufficiently well
covered.  Try Emoji (something in the range U+1F600 to U+1F64F), which
was the trigger for this bug and my changes, or Enclosed Alphanumerics
(U+1F100 to U+1F1FFF), for example.  Or even Supplemental Punctuation
(u+2E00 to u+2E7F) or Currency Symbols (u+20A0 to u+20CF).  What I see
on my system is that several fonts claim coverage, but typically
support just a few characters, sometimes just one.  That's the problem
I was trying to avoid.

> and on these environments the cure seems to be worse than the
> disease.  Is there some way we can heuristically tell whether we're
> in an environment where glyphs are often not supported?

It's not a matter of the environment, it's the matter of which fonts
are installed.  And sometimes, more installed fonts is worse, because
some fonts claim support for Unicode ranges where their actual
coverage is very scarce, tempting Emacs to try using them instead of
better ones.

> For example, can we convert a sample glyph or two to a bitmap and
> see whether it looks like a boxed hex code?  (Just thinking out
> loud.)

That's not necessary: Emacs can know whether a font supports a
character without any analysis of the resultant bitmap (see
font_has_char).  The problem is that Emacs tries very hard not to open
a font, and does all its font search without actually opening any
fonts.  By contrast, what you suggest is only possible once we open a
font.

I think the basic idea to augment the default fontset is correct, it
just needs to be fine-tuned to not mess up users' most popular setups.

> > I don't think it's a good idea to
> > go back to the previous arrangement where any font that claimed
> > iso10646-1 support would be considered as covering symbols and
> > punctuation well, because that means restoring the problems I tried to
> > fix in the first place.
> 
> In that case I don't understand why emacs -Q -font 
> -misc-fixed-medium-r-semicondensed--13-120-75-75-c-60-iso10646-1 works now.  If 
> Emacs is supposed to prefer Symbola to other fonts when displaying symbols, why 
> isn't it using Symbola in this case?

Because -misc-fixed-medium-r-semicondensed--13-120-75-75-c-60-iso10646-1
is different from
-misc-fixed-medium-r-semicondensed--13-120-75-75-c-60-iso8859-1, I
think.  The current code on master already does use the default font
if it supports these characters, but
-misc-fixed-medium-r-semicondensed--13-120-75-75-c-60-iso8859-1
doesn't, evidently.  So Emacs tries to look for another font.  Before
my changes it would find
-misc-fixed-medium-r-semicondensed--13-120-75-75-c-60-iso10646-1;
after them, it tries Symbola first.

bug#20727: 24.5; Font fallback doesn't work for the Emoji range

[Eli Zaretskii]

> Date: Sat, 13 Jun 2015 10:10:05 -0700
> From: Paul Eggert <eggert@cs.ucla.edu>
> CC: v.schneidermann@gmail.com, andrewjmoreton@gmail.com, 
>  20727@debbugs.gnu.org
> 
> Eli Zaretskii wrote:
> > Actually, an even better solution might be this: we explicitly specify
> > fixed-medium font with iso10646-1 registry for these popular
> > characters_before_  Symbola.  This is justified, I think, since the
> > standard-fontset-spec on X specifies a fixed-medium font, so fit is
> > iso10646-1 variant supports these commonly-used punctuation
> > characters, it is a better candidate than Symbola, while the latter
> > will still be used as fallback.
> >
> > WDYT?  Could you try this on your system and see if it work?
> 
> I'd be happy to try, but I'm afraid this is an area I don't know well

Neither do I, so please bear with me.

> and I 
> don't see why the existing code doesn't already do what you'r suggesting. 
> setup-default-fontset invokes (set-fontset-font "fontset-default" ... "Symbola" 
> nil 'prepend), and later invokes (set-fontset-font "fontset-default" nil '(nil . 
> "iso10646-1") nil 'prepend), so doesn't that mean that the iso10646-1 fonts are 
> prepended before Symbola and so should take priority over Symbola already?

No, not according to my understanding.  The latter part is fallback
for when nothing is specified as either the script or the character
range, which is not the case we are discussing.

> Anyway, If you can send me a patch along the lines you're thinking, I can try it.

Here's an example of the 1st idea.  If it works well, we can exempt
more ranges from Symbola (if you could tell me where to download the
fixed-medium font, I could look at its coverage to decide which other
ranges to exempt):

diff --git a/lisp/international/fontset.el b/lisp/international/fontset.el
index 696940e..cb32daa 100644
--- a/lisp/international/fontset.el
+++ b/lisp/international/fontset.el
@@ -697,7 +697,11 @@ (defun setup-default-fontset ()
   ;; covered well by Symbola.
   (dolist (symbol-subgroup
            '((#x0250 . #x02AF)	;; IPA Extensions
-             (#x2000 . #x206F)	;; General Punctuation
+             (#x2000 . #x2012)	;; General Punctuation
+             (#x2015 . #x2017)
+             (#x201F . #x202F)
+             (#x2031 . #x2038)
+             (#x203B . #x206F)
              (#x2070 . #x209F)	;; Superscripts and Subscripts
              (#x20A0 . #x20CF)	;; Currency Symbols
              (#x2100 . #x214F)	;; Letterlike Symbols

The second idea is to add

  (set-fontset-font "fontset-default" '(#x2013 . #x2014)
                    "-*-fixed-medium-r-*-*-*-*-*-*-*-*-iso10646-1"
		    nil 'prepend)
  (set-fontset-font "fontset-default" '(#x2018 . #x201E)
                    "-*-fixed-medium-r-*-*-*-*-*-*-*-*-iso10646-1"
		    nil 'prepend)
  (set-fontset-font "fontset-default" '(#x2039 . #x203A)
                    "-*-fixed-medium-r-*-*-*-*-*-*-*-*-iso10646-1"
		    nil 'prepend)

(and similarly for other ranges where that font has coverage) to
setup-default-fontset _after_ the Symbola portion.

bug#20727: 24.5; Font fallback doesn't work for the Emoji range

[Paul Eggert]

Eli Zaretskii wrote:

> I think you are judging by characters that are sufficiently well
> covered.  Try Emoji (something in the range U+1F600 to U+1F64F), which
> was the trigger for this bug and my changes, or Enclosed Alphanumerics
> (U+1F100 to U+1F1FFF), for example.  Or even Supplemental Punctuation
> (u+2E00 to u+2E7F) or Currency Symbols (u+20A0 to u+20CF).  What I see
> on my system is that several fonts claim coverage, but typically
> support just a few characters, sometimes just one.  That's the problem
> I was trying to avoid.

I tried the 8 specific characters that you mentioned (the endpoints of the 
ranges) with emacs -Q, and found that in my environment Symbola looked way 
better with U+1F600, U+1F64F, U+1F100 (where older Emacs just displays hex codes 
in boxes), that Symbola looks a bit worse with U+2E00 and U+20A0 (where older 
Emacs uses FreeSerif which better matches the FreeSerif characters elsewhere in 
the buffer), and that both fonts look bad (hex boxes) with U+1F1FFF, U+2E7F, 
U+20CF (as they're unassigned).

bug#20727: 24.5; Font fallback doesn't work for the Emoji range

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 485 bytes --]

I tried both suggestions, and I'm afraid they didn't work.  That is, I used the 
attached patch and ran "emacs -Q -r -font 
-misc-fixed-medium-r-semicondensed--0-0-75-75-c-0-iso8859-1", and Emacs 
displayed U+2018 LEFT SINGLE QUOTATION MARK using 
x:-unknown-Symbola-normal-normal-semicondensed-*-13-*-*-*-*-0-iso10646-1 
(#x394), which is almost unreadable compared to the old way which displayed it 
using x:-misc-fixed-medium-r-semicondensed--13-120-75-75-c-60-iso10646-1 (#x2018).

[-- Attachment #2: ideas.txt --]
[-- Type: text/plain, Size: 1435 bytes --]

diff --git a/lisp/international/fontset.el b/lisp/international/fontset.el
index 696940e..7c74f0e 100644
--- a/lisp/international/fontset.el
+++ b/lisp/international/fontset.el
@@ -698,6 +698,11 @@
   (dolist (symbol-subgroup
            '((#x0250 . #x02AF)	;; IPA Extensions
              (#x2000 . #x206F)	;; General Punctuation
+             (#x2000 . #x2012)	;; General Punctuation
+             (#x2015 . #x2017)
+             (#x201F . #x202F)
+             (#x2031 . #x2038)
+             (#x203B . #x206F)
              (#x2070 . #x209F)	;; Superscripts and Subscripts
              (#x20A0 . #x20CF)	;; Currency Symbols
              (#x2100 . #x214F)	;; Letterlike Symbols
@@ -738,6 +743,16 @@
   (set-fontset-font "fontset-default" '(#x2500 . #x259F)
                     "FreeMono" nil 'prepend)
 
+  (set-fontset-font "fontset-default" '(#x2013 . #x2014)
+                    "-*-fixed-medium-r-*-*-*-*-*-*-*-*-iso10646-1"
+		    nil 'prepend)
+  (set-fontset-font "fontset-default" '(#x2018 . #x201E)
+                    "-*-fixed-medium-r-*-*-*-*-*-*-*-*-iso10646-1"
+		    nil 'prepend)
+  (set-fontset-font "fontset-default" '(#x2039 . #x203A)
+                    "-*-fixed-medium-r-*-*-*-*-*-*-*-*-iso10646-1"
+		    nil 'prepend)
+
   ;; Append CJK fonts for characters other than han, kana, cjk-misc.
   ;; Append fonts for scripts whose name is also a charset name.
   (let* ((data (build-default-fontset-data))

bug#20727: 24.5; Font fallback doesn't work for the Emoji range

[Eli Zaretskii]

> Date: Sat, 13 Jun 2015 11:47:07 -0700
> From: Paul Eggert <eggert@cs.ucla.edu>
> CC: v.schneidermann@gmail.com, andrewjmoreton@gmail.com, 
>  20727@debbugs.gnu.org

> I tried the 8 specific characters that you mentioned (the endpoints of the 
> ranges) with emacs -Q, and found that in my environment Symbola looked way 
> better with U+1F600, U+1F64F, U+1F100 (where older Emacs just displays hex codes 
> in boxes), that Symbola looks a bit worse with U+2E00 and U+20A0 (where older 
> Emacs uses FreeSerif which better matches the FreeSerif characters elsewhere in 
> the buffer)

How did you get FreeSerif characters elsewhere in the buffer?  Which
characters were those?

> both fonts look bad (hex boxes) with U+1F1FFF, U+2E7F, U+20CF (as
> they're unassigned).

Sorry, I mentioned the block limits without checking what parts are
assigned.  (Also, U+1F1FFF was a typo.)  What do you see for U+2E3F
and U+20BD?

Anyway, does the idea of selectively removing codepoints from where we
currently specify Symbola sound good, given these trials?

bug#20727: 24.5; Font fallback doesn't work for the Emoji range

[Eli Zaretskii]

> Date: Sat, 13 Jun 2015 12:02:25 -0700
> From: Paul Eggert <eggert@cs.ucla.edu>
> CC: v.schneidermann@gmail.com, andrewjmoreton@gmail.com, 
>  20727@debbugs.gnu.org
> 
> I tried both suggestions, and I'm afraid they didn't work.  That is, I used the 
> attached patch and ran "emacs -Q -r -font 
> -misc-fixed-medium-r-semicondensed--0-0-75-75-c-0-iso8859-1", and Emacs 
> displayed U+2018 LEFT SINGLE QUOTATION MARK using 
> x:-unknown-Symbola-normal-normal-semicondensed-*-13-*-*-*-*-0-iso10646-1 
> (#x394), which is almost unreadable compared to the old way which displayed it 
> using x:-misc-fixed-medium-r-semicondensed--13-120-75-75-c-60-iso10646-1 (#x2018).

You forgot to delete this line, which was part of the patch I proposed:

-             (#x2000 . #x206F)	;; General Punctuation

Without deleting it, the entire range of General Punctuation is still
covered by Symbola.

bug#20727: 24.5; Font fallback doesn't work for the Emoji range

[Paul Eggert]

Eli Zaretskii wrote:
> How did you get FreeSerif characters elsewhere in the buffer?  Which
> characters were those?

It was the other characters like "abc".  But I'm afraid I can't reproduce the 
effect now.  I've changed my default environment and perhaps that has affected 
things.

> What do you see for U+2E3F and U+20BD?

With 'emacs -Q' it's the same as in Emacs 24.4, namely, the Symbola glyphs.

> Anyway, does the idea of selectively removing codepoints from where we
> currently specify Symbola sound good, given these trials?

After trying it a bit I'm worried that this will be flaky.  Often the Symbola 
fonts are better (much better, if the default fonts lack the symbols), often 
worse (if both fonts have the symbols), and it all depends on a lot of settings. 
  Users will notice when the Symbola fonts make things worse for them.

Instead of using Symbola for all symbols, perhaps we should just use it for 
emoticons and other symbols known to be commonly bad.

It's too bad that we can't fall back on Symbola only when the font is missing 
the character.  I still don't understand things well enough to propose an 
implementation along those lines, though.

bug#20727: 24.5; Font fallback doesn't work for the Emoji range

[Eli Zaretskii]

> Date: Sat, 13 Jun 2015 14:19:42 -0700
> From: Paul Eggert <eggert@cs.ucla.edu>
> CC: v.schneidermann@gmail.com, andrewjmoreton@gmail.com, 
>  20727@debbugs.gnu.org
> 
> > Anyway, does the idea of selectively removing codepoints from where we
> > currently specify Symbola sound good, given these trials?
> 
> After trying it a bit I'm worried that this will be flaky.  Often the Symbola 
> fonts are better (much better, if the default fonts lack the symbols), often 
> worse (if both fonts have the symbols), and it all depends on a lot of settings. 

What settings are those?  The only ones I think are relevant are the
default fontset and the default font.  Are there any others?

> Instead of using Symbola for all symbols, perhaps we should just use it for 
> emoticons and other symbols known to be commonly bad.

That's what I intend to do next: leave Symbola only for symbols that
are not well covered by other fonts.  Similar to the patch I've shown
earlier.

> It's too bad that we can't fall back on Symbola only when the font is missing 
> the character.  I still don't understand things well enough to propose an 
> implementation along those lines, though.

It'd be great if you or someone else did.  My impression from reading
the (quite convoluted) code was that building font selection on
actually trying to see if each candidate font has a glyph for a
character will require a complete rewrite of the font-selection
logic.  And even then, we have no way of knowing if the glyph we get
looks good.

bug#20727: 24.5; Font fallback doesn't work for the Emoji range

[Eli Zaretskii]

> Date: Sun, 14 Jun 2015 05:46:03 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: v.schneidermann@gmail.com, andrewjmoreton@gmail.com, 20727@debbugs.gnu.org
> 
> > Instead of using Symbola for all symbols, perhaps we should just use it for 
> > emoticons and other symbols known to be commonly bad.
> 
> That's what I intend to do next: leave Symbola only for symbols that
> are not well covered by other fonts.  Similar to the patch I've shown
> earlier.

I tried to do this in commit fce59d4; please try it.  I don't have an
easy way of testing the change related to fixed-medium font, so please
try the characters in the blocks for which I added preference for that
font, and see that they are displayed on your system using
"-misc-fixed-medium-r-...-iso10646-1" and not "Symbola".

Thanks.

bug#20727: 24.5; Font fallback doesn't work for the Emoji range

[Paul Eggert]

Eli Zaretskii wrote:

>> After trying it a bit I'm worried that this will be flaky.  Often the Symbola
>> fonts are better (much better, if the default fonts lack the symbols), often
>> worse (if both fonts have the symbols), and it all depends on a lot of settings.
>
> What settings are those?  The only ones I think are relevant are the
> default fontset and the default font.  Are there any others?

I'm not sure, but I found that my ~/.emacs file, at the end, said 
"(custom-set-faces)" -- something I didn't put in there, but I suppose I ran 
"customize" at some point in the unremembered past, though I don't remember ever 
customizing fonts.  I suppose the settings installed by custom-set-faces, 
whatever they are, alter the fonts installed by the recent change, and this 
messes up my testing.  (Possibly they saved faces calculated *before* the recent 
change to Emacs, and custom-set-faces is trying to restore them?)  I will 
comment out the customization code before doing further testing, but these are 
the sorts of glitches that I fear will affect other users.

I took a look at your recent commit 2f09f8952489b5c90488faf66f71a4252aed5c2c and 
things are better on Ubuntu, thanks.  In my default environment it now uses the 
fixed font so the Symbola stuff doesn't get in the way.  I'll try it out on 
Fedora in a couple of days.

I did try it with emacs -Q to get the Ubuntu 15.04 default font (Ubuntu Mono), 
and found that recent Emacs in many places looks nicer and in some places not as 
good.  Focusing on the latter:

U+201F DOUBLE HIGH-REVERSED-9 QUOTATION MARK now displays as 
xft:-unknown-Symbola-normal-normal-semicondensed-*-17-*-*-*-*-0-iso10646-1 
(#x39B) which is spidery, whereas it formerly displayed as 
xft:-unknown-FreeMono-normal-normal-normal-*-17-*-*-*-m-0-iso10646-1 (#x8A8) 
which is more legible.

U+204F REVERSED SEMICOLON has a similar problem.

U+2047 DOUBLE QUESTION MARK
U+2048 QUESTION EXCLAMATION MARK
U+2049 EXCLAMATION QUESTION MARK
are too large in Symbola; the old FreeMono version was better.

The currency symbols look worse than before: they used to be constant-width 
(most of them anyway) and matched Ubuntu Mono better.  Perhaps we should leave 
them alone?

Thanks for doing all this -- it must have taken you quite some time.

bug#20727: 24.5; Font fallback doesn't work for the Emoji range

[Eli Zaretskii]

> Date: Sun, 14 Jun 2015 09:14:27 -0700
> From: Paul Eggert <eggert@cs.ucla.edu>
> CC: v.schneidermann@gmail.com, andrewjmoreton@gmail.com, 
>  20727@debbugs.gnu.org
> 
> I'm not sure, but I found that my ~/.emacs file, at the end, said 
> "(custom-set-faces)" -- something I didn't put in there, but I suppose I ran 
> "customize" at some point in the unremembered past, though I don't remember ever 
> customizing fonts.  I suppose the settings installed by custom-set-faces, 
> whatever they are, alter the fonts installed by the recent change, and this 
> messes up my testing.  (Possibly they saved faces calculated *before* the recent 
> change to Emacs, and custom-set-faces is trying to restore them?)  I will 
> comment out the customization code before doing further testing, but these are 
> the sorts of glitches that I fear will affect other users.

We will have to wait and see, I guess.  I hope the problems will not
be acute, mostly when someone already have fontset customizations, and
therefore can adapt.

> U+201F DOUBLE HIGH-REVERSED-9 QUOTATION MARK now displays as 
> xft:-unknown-Symbola-normal-normal-semicondensed-*-17-*-*-*-*-0-iso10646-1 
> (#x39B) which is spidery, whereas it formerly displayed as 
> xft:-unknown-FreeMono-normal-normal-normal-*-17-*-*-*-m-0-iso10646-1 (#x8A8) 
> which is more legible.
> 
> U+204F REVERSED SEMICOLON has a similar problem.
> 
> U+2047 DOUBLE QUESTION MARK
> U+2048 QUESTION EXCLAMATION MARK
> U+2049 EXCLAMATION QUESTION MARK
> are too large in Symbola; the old FreeMono version was better.

These are all in the General Punctuation block.  FreeMono covers only
about one third of the block (40 characters out of 111); Symbola
covers all of them.  It's a simple change to prefer FreeMono for those
characters that it supports (after all, FreeMono is a GNU font), but
won't that have adverse effect if some other punctuation characters,
unsupported by FreeMono, will have to be displayed?  You can try
something like

  (set-fontset-font "fontset-default" '(#x2047 . #x204B) "FreeMono")

and then try typing characters from this range and also a few outside
of it, but still between 2000..206F -- is the result acceptable?  It
looks weird on my system, but I think I'm less sensitive to these
issues, so I'm not sure about others.

> The currency symbols look worse than before: they used to be constant-width 
> (most of them anyway) and matched Ubuntu Mono better.  Perhaps we should leave 
> them alone?

Which font did they use before?

In general, most fonts support only a handful of characters in the
Currency Symbol block.  I left only the Euro sign, which is almost
universally supported, out of Symbola coverage.  I could add a few
more to the exempted codepoints, or indeed leave out the entire block,
but then we could risk boxes with hex code with some fonts.

> Thanks for doing all this -- it must have taken you quite some time.

You're welcome.  Yes, that's a lot of mundane work, but someone needs
to do it.

bug#20727: 24.5; Font fallback doesn't work for the Emoji range

[Paul Eggert]

Eli Zaretskii wrote:
> You can try
> something like
>
>    (set-fontset-font "fontset-default" '(#x2047 . #x204B) "FreeMono")
>
> and then try typing characters from this range and also a few outside
> of it, but still between 2000..206F -- is the result acceptable?  It
> looks weird on my system, but I think I'm less sensitive to these
> issues, so I'm not sure about others.

For those two characters FreeMono does look nicer.  There are a few other 
characters where perhaps some people would prefer FreeMono too.  It's not a big 
deal, though.  Perhaps we should just leave it that way until someone with real 
experience with those symbols (i.e., not me) needs to use them in real text.

>> The currency symbols look worse than before: they used to be constant-width
>> (most of them anyway) and matched Ubuntu Mono better.  Perhaps we should leave
>> them alone?
>
> Which font did they use before?

Kind of a mishmash; see below.  Ah, now I see which ones don't line up in the 
fixed-width font: it's the three at the end, which use Symbola.  It's this lack 
of lining-up that is one of the hassles with Symbola.

₠ 20a0 DejaVu Sans Mono
₡ 20a1 FreeMono
₢ 20a2 FreeMono
₣ 20a3 FreeMono
₤ 20a4 FreeMono
₥ 20a5 FreeMono
₦ 20a6 FreeMono
₧ 20a7 FreeMono
₨ 20a8 FreeMono
₩ 20a9 FreeMono
₪ 20aa FreeMono
₫ 20ab FreeMono
€ 20ac Ubuntu Mono
₭ 20ad Phetsarath OT
₮ 20ae Ubuntu Mono
₯ 20af FreeMono
₰ 20b0 FreeMono
₱ 20b1 FreeMono
₲ 20b2 FreeMono
₳ 20b3 FreeMono
₴ 20b4 Ubuntu Mono
₵ 20b5 FreeMono
₶ 20b6 FreeSerif
₷ 20b7 FreeSerif
₸ 20b8 FreeMono
₹ 20b9 Ubuntu Mono
₺ 20ba DejaVu Sans
₻ 20bb Symbola
₼ 20bc Symbola
₽ 20bd Symbola

bug#20727: 24.5; Font fallback doesn't work for the Emoji range

[Eli Zaretskii]

> Date: Sun, 14 Jun 2015 13:39:31 -0700
> From: Paul Eggert <eggert@cs.ucla.edu>
> CC: v.schneidermann@gmail.com, andrewjmoreton@gmail.com, 
>  20727@debbugs.gnu.org
> 
>     You can try
>     something like
> 
>        (set-fontset-font "fontset-default" '(#x2047 . #x204B) "FreeMono")
> 
>     and then try typing characters from this range and also a few outside
>     of it, but still between 2000..206F -- is the result acceptable?  It
>     looks weird on my system, but I think I'm less sensitive to these
>     issues, so I'm not sure about others.
> 
> For those two characters FreeMono does look nicer. There are a few other characters where perhaps some people would prefer FreeMono too. It's not a big deal, though. Perhaps we should just leave it that way until someone with real experience with those symbols (i.e., not me) needs to use them in real text.

I agree, let's leave these alone.

>         The currency symbols look worse than before: they used to be constant-width
>         (most of them anyway) and matched Ubuntu Mono better.  Perhaps we should leave
>         them alone?
> 
>     Which font did they use before?

Yes, the changes I committed yesterday as part of fce59d4 make
fixed-medium-...-iso10646-1 font be preferred for this block, and
according to my references most of the currency symbols are covered by
that font, but not by Ubuntu Mono.  So on Fedora I expect you to see
them displayed with fixed-medium.

> Kind of a mishmash; see below. Ah, now I see which ones don't line up in the fixed-width font: it's the three at the end, which use Symbola. It's this lack of lining-up that is one of the hassles with Symbola.

Unfortunately, monospaced fonts generally have very poor coverage of
the punctuation and symbols blocks, so we are stuck with
variable-pitch fonts for at least some of those.

I tried to improve the situation to some extent in 643e052, by
limiting the portion of the Currency Symbols range where we specify
Symbola to the symbols that are almost never covered by
general-purpose fonts (the ones beyond U+20B5), but that might mean
some users will see boxes with hex codes for a few currency symbols.
We shall see.