Order of fonts returned by list-fonts

Order of fonts returned by list-fonts

[Kévin Le Gouguec]

Hello Emacs,

My distro ships with Noto Color Emoji 13.1.  I have installed version 14
under ~/.local/share/fonts.  fc-match seems to prioritize the latter:

$ fc-match -f '%{fullname} %{file}\n' "Noto Color Emoji"
> Noto Color Emoji /home/peniblec/.local/share/fonts/NotoColorEmoji.ttf

Emacs, however, seems to favour the former:

(find-font (font-spec :family "Noto Color Emoji"))
> #<font-entity ftcrhb GOOG Noto\ Color\ Emoji nil iso10646-1 regular
> normal normal 0 nil 100 0 ((:font-entity
> "/usr/share/fonts/truetype/NotoColorEmoji.ttf" . 0))>

(list-fonts (font-spec :family "Noto Color Emoji"))
> (#<font-entity ftcrhb GOOG Noto\ Color\ Emoji nil iso10646-1 regular
>  normal normal 0 nil 100 0 ((:font-entity
>  "/usr/share/fonts/truetype/NotoColorEmoji.ttf" . 0))> 
>  #<font-entity ftcrhb GOOG Noto\ Color\ Emoji nil iso10646-1 regular
>  normal normal 0 nil 100 0 ((:font-entity
>  "/home/peniblec/.local/share/fonts/NotoColorEmoji.ttf" . 0))>)

Could this be considered a bug?  I didn't dig into the code yet, beside
looking at font_list_entities (which led me to restart with
EMACS_FONT_LOG set; unfortunately font-show-log doesn't have much to say
about how font files are searched).

I don't know if my expectation (that Emacs and fc-match should agree on
how files are prioritized) is warranted; I know that Emacs relies on
Fontconfig APIs to some degree, but I'm sure there is more complexity
lurking under the hood.

I figured I'd ask around if the above makes sense to someone before
putting my spelunking hat on :)


FWIW: I originally "solved" the problem by uninstalling the version
shipped by my distro; unfortunately, today's upgrade seems to have
brought it back, presumably because another package has started
depending on it.

FWIW²: the bigger-picture fix would probably be to send a patch to the
distro to migrate to Noto Color Emoji 14; I might get to that at some
point, but I still would like to understand what's going on with Emacs.


In GNU Emacs 29.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.31, cairo version 1.16.0)
 of 2022-01-01 built on amdahl30
Repository revision: 1d3020908b4e4ff398c3faed9321aa4932fbaad1
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12101003
System Description: openSUSE Tumbleweed

Configured using:
 'configure --with-cairo --with-gconf --with-xinput2'

Configured features:
ACL CAIRO DBUS FREETYPE GCONF GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ
JPEG JSON LCMS2 LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 M17N_FLT MODULES
NOTIFY INOTIFY PDUMPER PNG RSVG SECCOMP SOUND SQLITE3 THREADS TIFF
TOOLKIT_SCROLL_BARS WEBP X11 XDBE XIM XINPUT2 XPM GTK3 ZLIB

Important settings:
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=local
  locale-coding-system: utf-8-unix

Re: Order of fonts returned by list-fonts

[Eli Zaretskii]

> From: Kévin Le Gouguec <kevin.legouguec@gmail.com>
> Date: Sat, 08 Jan 2022 23:59:28 +0100

This list is not the right place for discussions where you question
the correctness of low-level font-handling code in Emacs.  Please
consider moving this to emacs-devel.

> My distro ships with Noto Color Emoji 13.1.  I have installed version 14
> under ~/.local/share/fonts.  fc-match seems to prioritize the latter:
> 
> $ fc-match -f '%{fullname} %{file}\n' "Noto Color Emoji"
> > Noto Color Emoji /home/peniblec/.local/share/fonts/NotoColorEmoji.ttf
> 
> Emacs, however, seems to favour the former:
> 
> (find-font (font-spec :family "Noto Color Emoji"))
> > #<font-entity ftcrhb GOOG Noto\ Color\ Emoji nil iso10646-1 regular
> > normal normal 0 nil 100 0 ((:font-entity
> > "/usr/share/fonts/truetype/NotoColorEmoji.ttf" . 0))>
> 
> (list-fonts (font-spec :family "Noto Color Emoji"))
> > (#<font-entity ftcrhb GOOG Noto\ Color\ Emoji nil iso10646-1 regular
> >  normal normal 0 nil 100 0 ((:font-entity
> >  "/usr/share/fonts/truetype/NotoColorEmoji.ttf" . 0))> 
> >  #<font-entity ftcrhb GOOG Noto\ Color\ Emoji nil iso10646-1 regular
> >  normal normal 0 nil 100 0 ((:font-entity
> >  "/home/peniblec/.local/share/fonts/NotoColorEmoji.ttf" . 0))>)
> 
> Could this be considered a bug?  I didn't dig into the code yet, beside
> looking at font_list_entities (which led me to restart with
> EMACS_FONT_LOG set; unfortunately font-show-log doesn't have much to say
> about how font files are searched).

As you've probably seen, font_list_entities calls the font driver's
'list' method, and then reverses the resulting list.  That is supposed
to produce the list of fonts in the same order in which the font
driver lists the fonts.  Does that mean that Fontconfig produces
matching fonts with the best/newest/user-local one the last?

> I don't know if my expectation (that Emacs and fc-match should agree on
> how files are prioritized) is warranted; I know that Emacs relies on
> Fontconfig APIs to some degree, but I'm sure there is more complexity
> lurking under the hood.

Emacs doesn't assign any priorities to the fonts returned by the font
driver due to the order in which they are returned; it sorts the list
of fonts by other criteria.  However, after sorting (if there's more
than one candidate), Emacs will use the first matching font.  What
happens in your case can only be established by stepping through the
code with a debugger.

Re: Order of fonts returned by list-fonts

[Kévin Le Gouguec]

[-- Attachment #1: Type: text/plain, Size: 4780 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

> This list is not the right place for discussions where you question
> the correctness of low-level font-handling code in Emacs.  Please
> consider moving this to emacs-devel.

Thanks, I'll try to tune my list-picking heuristic (it's pretty hairy,
but I think the fault lies in the bit that says "unsure if bug or not ⇒
help-gnu-emacs").

>> My distro ships with Noto Color Emoji 13.1.  I have installed version 14
>> under ~/.local/share/fonts.  fc-match seems to prioritize the latter:
>> 
>> $ fc-match -f '%{fullname} %{file}\n' "Noto Color Emoji"
>> > Noto Color Emoji /home/peniblec/.local/share/fonts/NotoColorEmoji.ttf
>> 
>> Emacs, however, seems to favour the former:
>> 
>> (find-font (font-spec :family "Noto Color Emoji"))
>> > #<font-entity ftcrhb GOOG Noto\ Color\ Emoji nil iso10646-1 regular
>> > normal normal 0 nil 100 0 ((:font-entity
>> > "/usr/share/fonts/truetype/NotoColorEmoji.ttf" . 0))>
>> 
>> (list-fonts (font-spec :family "Noto Color Emoji"))
>> > (#<font-entity ftcrhb GOOG Noto\ Color\ Emoji nil iso10646-1 regular
>> >  normal normal 0 nil 100 0 ((:font-entity
>> >  "/usr/share/fonts/truetype/NotoColorEmoji.ttf" . 0))> 
>> >  #<font-entity ftcrhb GOOG Noto\ Color\ Emoji nil iso10646-1 regular
>> >  normal normal 0 nil 100 0 ((:font-entity
>> >  "/home/peniblec/.local/share/fonts/NotoColorEmoji.ttf" . 0))>)
>> 
>> Could this be considered a bug?  I didn't dig into the code yet, beside
>> looking at font_list_entities (which led me to restart with
>> EMACS_FONT_LOG set; unfortunately font-show-log doesn't have much to say
>> about how font files are searched).
>
> As you've probably seen, font_list_entities calls the font driver's
> 'list' method, and then reverses the resulting list.  That is supposed
> to produce the list of fonts in the same order in which the font
> driver lists the fonts.  Does that mean that Fontconfig produces
> matching fonts with the best/newest/user-local one the last?

(Mmm.  Each time I've seen an nreverse (specifically, at the end of
font_list_entities and ftfont_list), I've assumed it was because we had
just iterated over something and accumulated results by Fcons'ing an
item with an accumulator, so we had "inverted" the order of the thing we
were iterating on, and nreverse restored the "original" order)

>                          Does that mean that Fontconfig produces
> matching fonts with the best/newest/user-local one the last?

I haven't fully figured out what Fontconfig does nor why yet; I'm
currently comparing what ftfont_list does vs. what fc-match does; the
latter's code is somewhat straightforward:

https://gitlab.freedesktop.org/fontconfig/fontconfig/-/raw/main/fc-match/fc-match.c

AFAICT (trying to keep it simple and high-level; apologies if I'm
eliding important stuff, or if I've overlooked something),

- fc-match.c creates a pattern, feeds it to FcConfigSubstitute and
  FcDefaultSubstitute, *then* to FcFontSort or FcFontMatch, and then
  creates an FcFontSet.

- ftfont_list creates an FcPattern, feeds it to FcFontList, and we use
  what that returns.

Not sure yet if the problem stems from not calling FcConfigSubstitute
and FcDefaultSubstitute (ftfont_resolve_generic_family calls those, but
IIUC only on temporary a FcPattern; there's also an #ifdef'd-out bit
that calls FcConfigSubstitute) or from FcFontList.

FWIW, I've attached snippets trying to reproduce various ways to use the
Fontconfig API:

- fontlist.c does more or less what ftfont_list does (I think),

- fontmatch.c and fontsort.c do what fc-match and fc-match -a do.

- fontsubstitute.c does like fontlist.c, but sneaks in calls to
  Fc*Substitute functions before calling FcFontList,

Results:

> fontlist
> got 2 fonts
> val 0: /usr/share/fonts/truetype/NotoColorEmoji.ttf
> val 1: /home/peniblec/.local/share/fonts/NotoColorEmoji.ttf
> 
> fontmatch
> got 1 fonts
> val 0: /home/peniblec/.local/share/fonts/NotoColorEmoji.ttf
> 
> fontsort
> got 3487 fonts
> val 0: /home/peniblec/.local/share/fonts/NotoColorEmoji.ttf
> val 1: /usr/share/fonts/truetype/NotoColorEmoji.ttf
> (3485 more elided)
> 
> fontsubstitute
> got 0 fonts

I need to read more documentation to understand (1) how all those
functions are supposed to work together (2) which of these exactly does
or does not pay attention to the configured order (3) how exactly is
this order configured: I'm very happy that fc-match seems to prioritize
stuff in ~/.local/share/fonts, but I haven't read fonts-conf(5)
attentively enough to explain why that is the case.

(In the meantime, if someone reading this sees an obvious way to fix
things, by all means, don't wait for me 🙃)


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: dump.h --]
[-- Type: text/x-chdr, Size: 621 bytes --]

#pragma once

#include <assert.h>
#include <stdio.h>

#include <fontconfig/fontconfig.h>

void dump_fontset(FcFontSet* fset)
{
        size_t max = 2;
        printf("got %d fonts\n", fset->nfont);

        for (size_t i=0; i<fset->nfont && i<max; i++) {
                FcChar8* resultval;
                assert(
                        FcPatternGetString(fset->fonts[i], FC_FILE, 0, &resultval)
                        == FcResultMatch
                );
                printf("val %zu: %s\n", i, resultval);
        }

        if (fset->nfont > max)
                printf("(%zu more elided)\n", fset->nfont-max);
}

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: fontlist.c --]
[-- Type: text/x-csrc, Size: 701 bytes --]

#include <assert.h>
#include <stdio.h>

#include <fontconfig/fontconfig.h>

#include "dump.h"

int main()
{
        FcResult result;
        assert(FcInit());

        FcPattern* search = FcPatternCreate(); assert(search);

        FcValue searchval = {
                .type = FcTypeString,
                        .u = {
                                .s = "Noto Color Emoji"
                        }
        };

        assert(
                FcPatternAdd(search, FC_FAMILY, searchval, FcFalse)
        );

        FcObjectSet* objs = FcObjectSetBuild(FC_FILE);
        assert(objs);

        FcFontSet* fset = FcFontList(NULL, search, objs);
        assert(fset);
        dump_fontset(fset);
}

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: fontmatch.c --]
[-- Type: text/x-csrc, Size: 931 bytes --]

#include <assert.h>
#include <stdio.h>

#include <fontconfig/fontconfig.h>

#include "dump.h"

int main()
{
        FcResult result;
        assert(FcInit());

        FcPattern* search = FcPatternCreate(); assert(search);

        FcValue searchval = {
                .type = FcTypeString,
                        .u = {
                                .s = "Noto Color Emoji"
                        }
        };

        assert(
                FcPatternAdd(search, FC_FAMILY, searchval, FcFalse)
        );

        FcObjectSet* objs = FcObjectSetBuild(FC_FILE);
        assert(objs);

        assert(FcConfigSubstitute(NULL, search, FcMatchPattern));
        FcDefaultSubstitute(search);

        FcFontSet* fset = FcFontSetCreate();
        assert(fset);
        FcPattern* match = FcFontMatch(NULL, search, &result);
        assert(result == FcResultMatch);
        FcFontSetAdd(fset, match);
        dump_fontset(fset);
}

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #5: fontsort.c --]
[-- Type: text/x-csrc, Size: 861 bytes --]

#include <assert.h>
#include <stdio.h>

#include <fontconfig/fontconfig.h>

#include "dump.h"

int main()
{
        FcResult result;
        assert(FcInit());

        FcPattern* search = FcPatternCreate(); assert(search);

        FcValue searchval = {
                .type = FcTypeString,
                        .u = {
                                .s = "Noto Color Emoji"
                        }
        };

        assert(
                FcPatternAdd(search, FC_FAMILY, searchval, FcFalse)
        );

        FcObjectSet* objs = FcObjectSetBuild(FC_FILE);
        assert(objs);

        assert(FcConfigSubstitute(NULL, search, FcMatchPattern));
        FcDefaultSubstitute(search);

        FcFontSet* fset = FcFontSort(0, search, FcFalse, NULL, &result);
        assert(result == FcResultMatch);
        assert(fset);
        dump_fontset(fset);
}

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #6: fontsubstitute.c --]
[-- Type: text/x-csrc, Size: 805 bytes --]

#include <assert.h>
#include <stdio.h>

#include <fontconfig/fontconfig.h>

#include "dump.h"

int main()
{
        FcResult result;
        assert(FcInit());

        FcPattern* search = FcPatternCreate(); assert(search);

        FcValue searchval = {
                .type = FcTypeString,
                        .u = {
                                .s = "Noto Color Emoji"
                        }
        };

        assert(
                FcPatternAdd(search, FC_FAMILY, searchval, FcFalse)
        );

        FcObjectSet* objs = FcObjectSetBuild(FC_FILE);
        assert(objs);

        assert(FcConfigSubstitute(NULL, search, FcMatchPattern));
        FcDefaultSubstitute(search);

        FcFontSet* fset = FcFontList(NULL, search, objs);
        assert(fset);
        dump_fontset(fset);
}

[-- Attachment #7: makefile --]
[-- Type: text/plain, Size: 237 bytes --]

progs = $(patsubst                              \
    %.c,%,$(wildcard *.c)                       \
)

.PHONY: run
run: $(progs)
	@for p in $(progs) ; do echo $$p ; ./$$p ; echo ; done

$(progs): %: %.c dump.h
	gcc $< -lfontconfig -o $@

Re: Order of fonts returned by list-fonts

[Eli Zaretskii]

> From: Kévin Le Gouguec <kevin.legouguec@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Sun, 09 Jan 2022 20:43:49 +0100
> 
> > As you've probably seen, font_list_entities calls the font driver's
> > 'list' method, and then reverses the resulting list.  That is supposed
> > to produce the list of fonts in the same order in which the font
> > driver lists the fonts.  Does that mean that Fontconfig produces
> > matching fonts with the best/newest/user-local one the last?
> 
> (Mmm.  Each time I've seen an nreverse (specifically, at the end of
> font_list_entities and ftfont_list), I've assumed it was because we had
> just iterated over something and accumulated results by Fcons'ing an
> item with an accumulator, so we had "inverted" the order of the thing we
> were iterating on, and nreverse restored the "original" order)

That's right.  Here's what ftfont.c does to generate the list of
matching fonts:

  for (i = 0; i < fontset->nfont; i++)
    {
      FcPattern *pat = fontset->fonts[i];
      FcChar8 *str;

      if (FcPatternGetString (pat, FC_FAMILY, 0, &str) == FcResultMatch)
	list = Fcons (intern ((char *) str), list);
    }