How do I set up font fallback in a robust way?

How do I set up font fallback in a robust way?

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 2960 bytes --]

Hi emacs-devel,

I'm cross-posting this from https://emacs.stackexchange.com/questions/17205 following Eli's suggestion; I couldn't figure this out from reading the manual.

TL;DR: What's a simple way to reliably say: use Ubuntu Mono as the default font, FreeMono for the characters unsupported by Ubuntu Mono, and Symbola for characters unsupported by both?

Since my main programming font does not cover all the mathematical symbols I need, I initially set up font fallback as shown below:

    (set-fontset-font t 'unicode (font-spec :name "FreeMono") nil 'append)
    (set-fontset-font t 'unicode (font-spec :name "Symbola") nil 'append)

Unfortunately this also changed the font for some of the character that my main font (Ubuntu Mono) supports, so I changed it to

    (set-fontset-font t 'unicode (font-spec :name "Ubuntu Mono") nil)
    (set-fontset-font t 'unicode (font-spec :name "FreeMono") nil 'append)
    (set-fontset-font t 'unicode (font-spec :name "Symbola") nil 'append)

If my understanding is correct, this should ensure that characters that Ubuntu Mono cannot handle are handled by FreeMono, unless FreeMono doesn't have them, in which case they should be displayed using Symbola. It is also my understanding that `t` does the same as `"fontset-default"` above.

Unfortunately, there were still cases where the right font wasn't selected; I found that changing to

    (set-fontset-font t 'unicode (font-spec :name "Ubuntu Mono") nil)
    (set-fontset-font t 'unicode (font-spec :name "FreeMono") nil 'append)
    (set-fontset-font t 'unicode (font-spec :name "Symbola") nil 'append)
    (set-fontset-font "fontset-startup" 'unicode (font-spec :name "Ubuntu Mono") nil)
    (set-fontset-font "fontset-startup" 'unicode (font-spec :name "FreeMono") nil 'append)
    (set-fontset-font "fontset-startup" 'unicode (font-spec :name "Symbola") nil 'append)

worked better, but not always: changing the font size using

    (set-face-attribute 'default nil :height some-size)

caused the fallbacks to be ignored, due to new fontsets being created.

My current solution is to do

    (set-fontset-font fontset 'unicode (font-spec :name "Ubuntu Mono") nil)
    (set-fontset-font fontset 'unicode (font-spec :name "FreeMono") nil 'append)
    (set-fontset-font fontset 'unicode (font-spec :name "Symbola") nil 'append)

on each fontset (`fontset-list`), after each font size change. This is cumbersome, and I have trouble imagining that it's the right solution.

What's the proper way to configure font fallback?

*Note*: for testing purposes, here are a few math characters: `ℕ𝓟⧺×≠≥≤±¬∨∧∃∀λ⟿⟹⊥⊤⊢`

*References*: Emacs manual on [fontsets](https://www.gnu.org/software/emacs/manual/html_node/emacs/Fontsets.html) and on [modifying fontsets](https://www.gnu.org/software/emacs/manual/html_node/emacs/Defining-Fontsets.html#Defining-Fontsets)

Cheers,
Clément.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: How do I set up font fallback in a robust way?

[Eli Zaretskii]

> From: Clément Pit--Claudel <clement.pit@gmail.com>
> Date: Sat, 9 Apr 2016 15:14:50 -0400
> 
> Since my main programming font does not cover all the mathematical symbols I need, I initially set up font fallback as shown below:
> 
>     (set-fontset-font t 'unicode (font-spec :name "FreeMono") nil 'append)
>     (set-fontset-font t 'unicode (font-spec :name "Symbola") nil 'append)

The default fontset already defines specific blocks to be handled by
these fonts, see fontset.el.  What problems, specifically, do you have
with the default setup and the current emacs-25 branch?

> What's the proper way to configure font fallback?

In general: find the characters you want to change and augment the
default fontset to use the font you want for them.

But let's first see the specific problems you have with the Emacs 25
defaults.  This issue got some attention a few months ago, so I'm
surprised to hear there are still problems in that area.

> *Note*: for testing purposes, here are a few math characters: `ℕ𝓟⧺×≠≥≤±¬∨∧∃∀λ⟿⟹⊥⊤⊢`

Which, if any, of those give you trouble with the default fontset?

Re: How do I set up font fallback in a robust way?

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 1679 bytes --]

On 04/09/2016 03:46 PM, Eli Zaretskii wrote:
> The default fontset already defines specific blocks to be handled by
> these fonts, see fontset.el.  What problems, specifically, do you have
> with the default setup and the current emacs-25 branch?

I get relatively inconsistent results in terms of which font is chosen to display a given symbol. The pair of fonts that I mentioned (Ubuntu Mono and Symbola) was just an example; my question is about configuring fallbacks for any given pair of fonts: it could be Source Code Pro and Asana Math instead of Ubuntu Mono and Symbola, for example.

>> What's the proper way to configure font fallback?
> 
> In general: find the characters you want to change and augment the
> default fontset to use the font you want for them.

Is it enough to augment the default fontset? Changing the :height of the 'default font creates a new fontset, which doesn't respect the fallbacks that I configure. Hence my using

  (dolist (ft (fontset-list))
    (set-fontset-font ft 'unicode (font-spec :name "Ubuntu Mono"))
    (set-fontset-font ft 'unicode (font-spec :name "Symbola") nil 'append))

> But let's first see the specific problems you have with the Emacs 25
> defaults.  This issue got some attention a few months ago, so I'm
> surprised to hear there are still problems in that area.
> 
>> *Note*: for testing purposes, here are a few math characters: `ℕ𝓟⧺×≠≥≤±¬∨∧∃∀λ⟿⟹⊥⊤⊢`
> 
> Which, if any, of those give you trouble with the default fontset?

𝓟 is displayed using `Latin Modern Math' in emacs -Q, it seems. In any case, Ubuntu Mono and Symbola were just two examples.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: How do I set up font fallback in a robust way?

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Clément Pit--Claudel <clement.pit@gmail.com>
> Date: Sat, 9 Apr 2016 16:01:27 -0400
> 
> On 04/09/2016 03:46 PM, Eli Zaretskii wrote:
> > The default fontset already defines specific blocks to be handled by
> > these fonts, see fontset.el.  What problems, specifically, do you have
> > with the default setup and the current emacs-25 branch?
> 
> I get relatively inconsistent results in terms of which font is chosen to display a given symbol.

I don't think I understand well enough what that means.  This issue
must be dealt with by presenting specific examples.

> The pair of fonts that I mentioned (Ubuntu Mono and Symbola) was just an example; my question is about configuring fallbacks for any given pair of fonts: it could be Source Code Pro and Asana Math instead of Ubuntu Mono and Symbola, for example.

I answered the general question already.  You can find examples of
that in fontset.el.

> >> What's the proper way to configure font fallback?
> > 
> > In general: find the characters you want to change and augment the
> > default fontset to use the font you want for them.
> 
> Is it enough to augment the default fontset? Changing the :height of the 'default font creates a new fontset, which doesn't respect the fallbacks that I configure.

Again, please provide specific examples, because I don't think I
understand how changing height could get in the way.

> >> *Note*: for testing purposes, here are a few math characters: `ℕ𝓟⧺×≠≥≤±¬∨∧∃∀λ⟿⟹⊥⊤⊢`
> > 
> > Which, if any, of those give you trouble with the default fontset?
> 
> 𝓟 is displayed using `Latin Modern Math' in emacs -Q, it seems.

Is that not okay for some reason?  If so, why?

Re: How do I set up font fallback in a robust way?

[Clément Pit--Claudel]

[-- Attachment #1.1: Type: text/plain, Size: 2899 bytes --]

On 04/09/2016 10:44 PM, Eli Zaretskii wrote:
>> Cc: emacs-devel@gnu.org From: Clément Pit--Claudel 
>> <clement.pit@gmail.com> Date: Sat, 9 Apr 2016 16:01:27 -0400
>> 
>> On 04/09/2016 03:46 PM, Eli Zaretskii wrote:
>>> The default fontset already defines specific blocks to be
>>> handled by these fonts, see fontset.el.  What problems,
>>> specifically, do you have with the default setup and the current
>>> emacs-25 branch?
>> 
>> I get relatively inconsistent results in terms of which font is 
>> chosen to display a given symbol.
> 
> I don't think I understand well enough what that means.  This issue 
> must be dealt with by presenting specific examples.

Sorry, I think I must not have described this problem properly in my initial message. Please let me find a good example.

In any case, here is something I came across while trying to produce one:

In emacs -Q, insert the following in the *scratch* buffer:

    (set-fontset-font "fontset-default" 'unicode (font-spec :name "Ubuntu Mono") nil)
    (set-fontset-font "fontset-default" 'unicode (font-spec :name "FreeMono") nil 'append)
    (set-fontset-font "fontset-default" 'unicode (font-spec :name "Symbola") nil 'append)

    (insert "ℕ𝓟⧺×≠≥≤±¬∨∧∃∀λ⟿⟹⊥⊤⊢")

Now run the lines one by one, and observe the "(insert ...)" line. Before running anything, it looks fine. It still looks fine after running the first line. Running the second line causes the 𝓟 to disappear; it is replaced by a square containing 01D4DF, and C-u C-x = says "display: no font available". Finally, running the third line fixes the 𝓟.

I don't understand how this is possible. How can adding FreeMono cause Emacs to think that no font is available to display the 𝓟?

>> Is it enough to augment the default fontset? Changing the :height 
>> of the 'default font creates a new fontset, which doesn't respect 
>> the fallbacks that I configure.
> 
> Again, please provide specific examples, because I don't think I 
> understand how changing height could get in the way.

Same here; I need to find a good example. I ran into issues that seemed to be connected with the fact that (set-face-attribute 'default nil :height 120) creates a new fontset, as reported by fontset-list:

    (fontset-list)
    => ("-unknown-Ubuntu Mono-normal-normal-normal-*-15-*-*-*-m-0-fontset-startup" "-*-fixed-medium-r-normal-*-16-*-*-*-*-*-fontset-standard" "-*-*-*-*-*-*-*-*-*-*-*-*-fontset-default")

    (set-face-attribute 'default nil :height 120)
    => nil

    (fontset-list)
    => ("-unknown-Ubuntu Mono-normal-normal-normal-*-16-*-*-*-m-0-fontset-auto1" "-unknown-Ubuntu Mono-normal-normal-normal-*-15-*-*-*-m-0-fontset-startup" "-*-fixed-medium-r-normal-*-16-*-*-*-*-*-fontset-standard" "-*-*-*-*-*-*-*-*-*-*-*-*-fontset-default")




Thanks for your help,
Clément.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: How do I set up font fallback in a robust way?

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Clément Pit--Claudel <clement.pit@gmail.com>
> Date: Sun, 10 Apr 2016 10:01:20 -0400
> 
> In emacs -Q, insert the following in the *scratch* buffer:
> 
>     (set-fontset-font "fontset-default" 'unicode (font-spec :name "Ubuntu Mono") nil)
>     (set-fontset-font "fontset-default" 'unicode (font-spec :name "FreeMono") nil 'append)
>     (set-fontset-font "fontset-default" 'unicode (font-spec :name "Symbola") nil 'append)
> 
>     (insert "ℕ𝓟⧺×≠≥≤±¬∨∧∃∀λ⟿⟹⊥⊤⊢")
> 
> Now run the lines one by one, and observe the "(insert ...)" line. Before running anything, it looks fine. It still looks fine after running the first line. Running the second line causes the 𝓟 to disappear; it is replaced by a square containing 01D4DF, and C-u C-x = says "display: no font available". Finally, running the third line fixes the 𝓟.
> 
> I don't understand how this is possible. How can adding FreeMono cause Emacs to think that no font is available to display the 𝓟?

If this surprises you, your mental model of how Emacs looks for a
suitable font needs to be updated. ;-)

I'm guessing you expect Emacs to test whether each font can actually
display a character.  But doing so would be prohibitively slow,
because it requires opening each font, an expensive operation.  There
are hundreds of fonts installed on any given system, and opening all
of them one by one takes a long time and considerably slows redisplay.

So Emacs uses shortcuts: it examines the font meta-data, where a font
declares which blocks of Unicode it supports, and picks up the first
font that claims support for the block of the character.  But if a
font claims support for a certain block, but has no glyphs for the
specific character we need to display, we lose.  Coupled with the fact
that you have no control on the order in which Emacs looks up fonts,
this is why adding a font to a fontset could actually make things
worse.

For this reason, just lumping fonts indiscriminately into the fontset
is not a good idea.  Instead, you should examine the character ranges
where you have problems, and add fonts in a way that instructs Emacs
to use each font only in the range of characters where you know it
will produce good results.  And since "good results" is largely in the
eyes of the beholder, this can only approximately be done in the
default settings, the rest must be done by the end user.

> Same here; I need to find a good example. I ran into issues that seemed to be connected with the fact that (set-face-attribute 'default nil :height 120) creates a new fontset, as reported by fontset-list:
> 
>     (fontset-list)
>     => ("-unknown-Ubuntu Mono-normal-normal-normal-*-15-*-*-*-m-0-fontset-startup" "-*-fixed-medium-r-normal-*-16-*-*-*-*-*-fontset-standard" "-*-*-*-*-*-*-*-*-*-*-*-*-fontset-default")
> 
>     (set-face-attribute 'default nil :height 120)
>     => nil
> 
>     (fontset-list)
>     => ("-unknown-Ubuntu Mono-normal-normal-normal-*-16-*-*-*-m-0-fontset-auto1" "-unknown-Ubuntu Mono-normal-normal-normal-*-15-*-*-*-m-0-fontset-startup" "-*-fixed-medium-r-normal-*-16-*-*-*-*-*-fontset-standard" "-*-*-*-*-*-*-*-*-*-*-*-*-fontset-default")

It just added the fonts for the larger height, but they are fonts from
the same families, so I see no issues here: if the original fontset
supported the characters you want, the updated one will, too.