Re: Order of fonts returned by list-fonts

Re: Order of fonts returned by list-fonts

[Kévin Le Gouguec]

[-- Attachment #1: Type: text/plain, Size: 4780 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

> This list is not the right place for discussions where you question
> the correctness of low-level font-handling code in Emacs.  Please
> consider moving this to emacs-devel.

Thanks, I'll try to tune my list-picking heuristic (it's pretty hairy,
but I think the fault lies in the bit that says "unsure if bug or not ⇒
help-gnu-emacs").

>> My distro ships with Noto Color Emoji 13.1.  I have installed version 14
>> under ~/.local/share/fonts.  fc-match seems to prioritize the latter:
>> 
>> $ fc-match -f '%{fullname} %{file}\n' "Noto Color Emoji"
>> > Noto Color Emoji /home/peniblec/.local/share/fonts/NotoColorEmoji.ttf
>> 
>> Emacs, however, seems to favour the former:
>> 
>> (find-font (font-spec :family "Noto Color Emoji"))
>> > #<font-entity ftcrhb GOOG Noto\ Color\ Emoji nil iso10646-1 regular
>> > normal normal 0 nil 100 0 ((:font-entity
>> > "/usr/share/fonts/truetype/NotoColorEmoji.ttf" . 0))>
>> 
>> (list-fonts (font-spec :family "Noto Color Emoji"))
>> > (#<font-entity ftcrhb GOOG Noto\ Color\ Emoji nil iso10646-1 regular
>> >  normal normal 0 nil 100 0 ((:font-entity
>> >  "/usr/share/fonts/truetype/NotoColorEmoji.ttf" . 0))> 
>> >  #<font-entity ftcrhb GOOG Noto\ Color\ Emoji nil iso10646-1 regular
>> >  normal normal 0 nil 100 0 ((:font-entity
>> >  "/home/peniblec/.local/share/fonts/NotoColorEmoji.ttf" . 0))>)
>> 
>> Could this be considered a bug?  I didn't dig into the code yet, beside
>> looking at font_list_entities (which led me to restart with
>> EMACS_FONT_LOG set; unfortunately font-show-log doesn't have much to say
>> about how font files are searched).
>
> As you've probably seen, font_list_entities calls the font driver's
> 'list' method, and then reverses the resulting list.  That is supposed
> to produce the list of fonts in the same order in which the font
> driver lists the fonts.  Does that mean that Fontconfig produces
> matching fonts with the best/newest/user-local one the last?

(Mmm.  Each time I've seen an nreverse (specifically, at the end of
font_list_entities and ftfont_list), I've assumed it was because we had
just iterated over something and accumulated results by Fcons'ing an
item with an accumulator, so we had "inverted" the order of the thing we
were iterating on, and nreverse restored the "original" order)

>                          Does that mean that Fontconfig produces
> matching fonts with the best/newest/user-local one the last?

I haven't fully figured out what Fontconfig does nor why yet; I'm
currently comparing what ftfont_list does vs. what fc-match does; the
latter's code is somewhat straightforward:

https://gitlab.freedesktop.org/fontconfig/fontconfig/-/raw/main/fc-match/fc-match.c

AFAICT (trying to keep it simple and high-level; apologies if I'm
eliding important stuff, or if I've overlooked something),

- fc-match.c creates a pattern, feeds it to FcConfigSubstitute and
  FcDefaultSubstitute, *then* to FcFontSort or FcFontMatch, and then
  creates an FcFontSet.

- ftfont_list creates an FcPattern, feeds it to FcFontList, and we use
  what that returns.

Not sure yet if the problem stems from not calling FcConfigSubstitute
and FcDefaultSubstitute (ftfont_resolve_generic_family calls those, but
IIUC only on temporary a FcPattern; there's also an #ifdef'd-out bit
that calls FcConfigSubstitute) or from FcFontList.

FWIW, I've attached snippets trying to reproduce various ways to use the
Fontconfig API:

- fontlist.c does more or less what ftfont_list does (I think),

- fontmatch.c and fontsort.c do what fc-match and fc-match -a do.

- fontsubstitute.c does like fontlist.c, but sneaks in calls to
  Fc*Substitute functions before calling FcFontList,

Results:

> fontlist
> got 2 fonts
> val 0: /usr/share/fonts/truetype/NotoColorEmoji.ttf
> val 1: /home/peniblec/.local/share/fonts/NotoColorEmoji.ttf
> 
> fontmatch
> got 1 fonts
> val 0: /home/peniblec/.local/share/fonts/NotoColorEmoji.ttf
> 
> fontsort
> got 3487 fonts
> val 0: /home/peniblec/.local/share/fonts/NotoColorEmoji.ttf
> val 1: /usr/share/fonts/truetype/NotoColorEmoji.ttf
> (3485 more elided)
> 
> fontsubstitute
> got 0 fonts

I need to read more documentation to understand (1) how all those
functions are supposed to work together (2) which of these exactly does
or does not pay attention to the configured order (3) how exactly is
this order configured: I'm very happy that fc-match seems to prioritize
stuff in ~/.local/share/fonts, but I haven't read fonts-conf(5)
attentively enough to explain why that is the case.

(In the meantime, if someone reading this sees an obvious way to fix
things, by all means, don't wait for me 🙃)


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: dump.h --]
[-- Type: text/x-chdr, Size: 621 bytes --]

#pragma once

#include <assert.h>
#include <stdio.h>

#include <fontconfig/fontconfig.h>

void dump_fontset(FcFontSet* fset)
{
        size_t max = 2;
        printf("got %d fonts\n", fset->nfont);

        for (size_t i=0; i<fset->nfont && i<max; i++) {
                FcChar8* resultval;
                assert(
                        FcPatternGetString(fset->fonts[i], FC_FILE, 0, &resultval)
                        == FcResultMatch
                );
                printf("val %zu: %s\n", i, resultval);
        }

        if (fset->nfont > max)
                printf("(%zu more elided)\n", fset->nfont-max);
}

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: fontlist.c --]
[-- Type: text/x-csrc, Size: 701 bytes --]

#include <assert.h>
#include <stdio.h>

#include <fontconfig/fontconfig.h>

#include "dump.h"

int main()
{
        FcResult result;
        assert(FcInit());

        FcPattern* search = FcPatternCreate(); assert(search);

        FcValue searchval = {
                .type = FcTypeString,
                        .u = {
                                .s = "Noto Color Emoji"
                        }
        };

        assert(
                FcPatternAdd(search, FC_FAMILY, searchval, FcFalse)
        );

        FcObjectSet* objs = FcObjectSetBuild(FC_FILE);
        assert(objs);

        FcFontSet* fset = FcFontList(NULL, search, objs);
        assert(fset);
        dump_fontset(fset);
}

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #4: fontmatch.c --]
[-- Type: text/x-csrc, Size: 931 bytes --]

#include <assert.h>
#include <stdio.h>

#include <fontconfig/fontconfig.h>

#include "dump.h"

int main()
{
        FcResult result;
        assert(FcInit());

        FcPattern* search = FcPatternCreate(); assert(search);

        FcValue searchval = {
                .type = FcTypeString,
                        .u = {
                                .s = "Noto Color Emoji"
                        }
        };

        assert(
                FcPatternAdd(search, FC_FAMILY, searchval, FcFalse)
        );

        FcObjectSet* objs = FcObjectSetBuild(FC_FILE);
        assert(objs);

        assert(FcConfigSubstitute(NULL, search, FcMatchPattern));
        FcDefaultSubstitute(search);

        FcFontSet* fset = FcFontSetCreate();
        assert(fset);
        FcPattern* match = FcFontMatch(NULL, search, &result);
        assert(result == FcResultMatch);
        FcFontSetAdd(fset, match);
        dump_fontset(fset);
}

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #5: fontsort.c --]
[-- Type: text/x-csrc, Size: 861 bytes --]

#include <assert.h>
#include <stdio.h>

#include <fontconfig/fontconfig.h>

#include "dump.h"

int main()
{
        FcResult result;
        assert(FcInit());

        FcPattern* search = FcPatternCreate(); assert(search);

        FcValue searchval = {
                .type = FcTypeString,
                        .u = {
                                .s = "Noto Color Emoji"
                        }
        };

        assert(
                FcPatternAdd(search, FC_FAMILY, searchval, FcFalse)
        );

        FcObjectSet* objs = FcObjectSetBuild(FC_FILE);
        assert(objs);

        assert(FcConfigSubstitute(NULL, search, FcMatchPattern));
        FcDefaultSubstitute(search);

        FcFontSet* fset = FcFontSort(0, search, FcFalse, NULL, &result);
        assert(result == FcResultMatch);
        assert(fset);
        dump_fontset(fset);
}

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #6: fontsubstitute.c --]
[-- Type: text/x-csrc, Size: 805 bytes --]

#include <assert.h>
#include <stdio.h>

#include <fontconfig/fontconfig.h>

#include "dump.h"

int main()
{
        FcResult result;
        assert(FcInit());

        FcPattern* search = FcPatternCreate(); assert(search);

        FcValue searchval = {
                .type = FcTypeString,
                        .u = {
                                .s = "Noto Color Emoji"
                        }
        };

        assert(
                FcPatternAdd(search, FC_FAMILY, searchval, FcFalse)
        );

        FcObjectSet* objs = FcObjectSetBuild(FC_FILE);
        assert(objs);

        assert(FcConfigSubstitute(NULL, search, FcMatchPattern));
        FcDefaultSubstitute(search);

        FcFontSet* fset = FcFontList(NULL, search, objs);
        assert(fset);
        dump_fontset(fset);
}

[-- Attachment #7: makefile --]
[-- Type: text/plain, Size: 237 bytes --]

progs = $(patsubst                              \
    %.c,%,$(wildcard *.c)                       \
)

.PHONY: run
run: $(progs)
	@for p in $(progs) ; do echo $$p ; ./$$p ; echo ; done

$(progs): %: %.c dump.h
	gcc $< -lfontconfig -o $@

Re: Order of fonts returned by list-fonts

[Eli Zaretskii]

> From: Kévin Le Gouguec <kevin.legouguec@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Sun, 09 Jan 2022 20:43:49 +0100
> 
> > As you've probably seen, font_list_entities calls the font driver's
> > 'list' method, and then reverses the resulting list.  That is supposed
> > to produce the list of fonts in the same order in which the font
> > driver lists the fonts.  Does that mean that Fontconfig produces
> > matching fonts with the best/newest/user-local one the last?
> 
> (Mmm.  Each time I've seen an nreverse (specifically, at the end of
> font_list_entities and ftfont_list), I've assumed it was because we had
> just iterated over something and accumulated results by Fcons'ing an
> item with an accumulator, so we had "inverted" the order of the thing we
> were iterating on, and nreverse restored the "original" order)

That's right.  Here's what ftfont.c does to generate the list of
matching fonts:

  for (i = 0; i < fontset->nfont; i++)
    {
      FcPattern *pat = fontset->fonts[i];
      FcChar8 *str;

      if (FcPatternGetString (pat, FC_FAMILY, 0, &str) == FcResultMatch)
	list = Fcons (intern ((char *) str), list);
    }