script-representative-chars vs incomplete fonts

script-representative-chars vs incomplete fonts

[Kévin Le Gouguec]

Hello,

IIUC, fontset.el sets up script-representative-chars so that any font
providing U+1D49C "𝒜" can be used for the whole "mathematical-script"
range (U+1D49C–U+1D4CF "𝒜–𝓏").

On my system, I happen to have

- one font (KpMath) that provides U+1D49C–U+1D4B5 (the "capital" chars),
  but not U+1D4B6–U+1D4CF (the "small" chars),

- another font (DejaVu Math TeX Gyre) that provides both sub-ranges[1].

As luck would have it, when trying to display a char in this range,
Emacs stumbles on KpMath first, and thus fails to display the "small"
sub-range (𝒶–𝓏): I get the dreaded "hex-in-a-box" placeholder for these
"small" codepoints.

The situation is not irremediable; I've thought of at least 3
workarounds[2].  I do wonder though if this situation is something Emacs
should guard against by default, and if so, with what heuristics?

E.g. in fontset.el, would it be overkill to break down the 52-char
ranges in the math-subgroup loop into 2 26-char ranges in order to
always add the "SMALL A" codepoints to script-representative-chars?

(The current loop builds vectors though; maybe we'd want lists?)

I'm sending this to help-gnu-emacs because I'm mostly puzzled; I don't
know if this problem is noteworthy enough to warrant a bug report.

Thoughts welcome.


[1] … minus, of course, the codepoints that do not exist because they
    are redundant with those in the "Letterlike Symbols" block.

[2] 1 and 2 work; haven't tried 3 yet:

    1. (push #x1D4CF (alist-get 'mathematical-script
                                script-representative-chars))
    2. (set-fontset-font
        t 'mathematical "DejaVu Math TeX Gyre" nil 'prepend)
    3. Uninstall KpMath.

Re: script-representative-chars vs incomplete fonts

[Eli Zaretskii]

> From: Kévin Le Gouguec <kevin.legouguec@gmail.com>
> Date: Sun, 12 Sep 2021 00:09:21 +0200
> 
> IIUC, fontset.el sets up script-representative-chars so that any font
> providing U+1D49C "𝒜" can be used for the whole "mathematical-script"
> range (U+1D49C–U+1D4CF "𝒜–𝓏").

No, script-representative-chars is used to _reject_ fonts that don't
suit our needs.  The candidate fonts should be selected by your font
backend, and selected wisely.

> On my system, I happen to have
> 
> - one font (KpMath) that provides U+1D49C–U+1D4B5 (the "capital" chars),
>   but not U+1D4B6–U+1D4CF (the "small" chars),
> 
> - another font (DejaVu Math TeX Gyre) that provides both sub-ranges[1].
> 
> As luck would have it, when trying to display a char in this range,
> Emacs stumbles on KpMath first

It's not Emacs that stumbles on that font, it's your font backend
(probably fontconfig?).  Emacs just verifies that the candidate fonts
returned by the backend satisfy the requirements, and uses the first
font that does.  So this might mean a bug report for fontconfig and/or
the font designers is in order.

> The situation is not irremediable; I've thought of at least 3
> workarounds[2].  I do wonder though if this situation is something Emacs
> should guard against by default, and if so, with what heuristics?

If there are strange fonts that support only portions of that Unicode
block, the easiest solution is to add more representative characters.

> E.g. in fontset.el, would it be overkill to break down the 52-char
> ranges in the math-subgroup loop into 2 26-char ranges in order to
> always add the "SMALL A" codepoints to script-representative-chars?

What would be the purpose of breaking that range?
script-representative-chars can specify multiple characters per
script, see its doc string.

>     1. (push #x1D4CF (alist-get 'mathematical-script
>                                 script-representative-chars))
>     2. (set-fontset-font
>         t 'mathematical "DejaVu Math TeX Gyre" nil 'prepend)
>     3. Uninstall KpMath.

I'd suggest to do 3 regardless.  A font that supports only part of a
block is a bad font to have.

Emacs should probably do the equivalent of 1, and the solution for
end-users is 2, of course; messing with script-representative-chars is
not recommended.

Re: script-representative-chars vs incomplete fonts

[Kévin Le Gouguec]

[-- Attachment #1: Type: text/plain, Size: 1615 bytes --]

Thanks for your insights, Eli!  Some follow-up thoughts & questions.

Eli Zaretskii <eliz@gnu.org> writes:

>> On my system, I happen to have
>> 
>> - one font (KpMath) that provides U+1D49C–U+1D4B5 (the "capital" chars),
>>   but not U+1D4B6–U+1D4CF (the "small" chars),
>> 
>> - another font (DejaVu Math TeX Gyre) that provides both sub-ranges[1].
>> 
>> As luck would have it, when trying to display a char in this range,
>> Emacs stumbles on KpMath first
>
> It's not Emacs that stumbles on that font, it's your font backend
> (probably fontconfig?).  Emacs just verifies that the candidate fonts
> returned by the backend satisfy the requirements, and uses the first
> font that does.  So this might mean a bug report for fontconfig and/or
> the font designers is in order.

Right; "as luck would have it", fontconfig returns KpMath rather than
DejaVu Math TeX Gyre; that's because Emacs only asks for U+1D49C, if I'm
reading ftfont.c's use of Vscript_representative_chars correctly?

So I'm not sure there's a bug on fontconfig's end?  FWIW the CLI tools
seem to provide accurate information[1].

>> E.g. in fontset.el, would it be overkill to break down the 52-char
>> ranges in the math-subgroup loop into 2 26-char ranges in order to
>> always add the "SMALL A" codepoints to script-representative-chars?
>
> What would be the purpose of breaking that range?
> script-representative-chars can specify multiple characters per
> script, see its doc string.

Right now script-representative-chars contains this:

> (mathematical-script 119964)

With this patch:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 667 bytes --]

diff --git a/lisp/international/fontset.el b/lisp/international/fontset.el
index 3deaff9677..461c8f830a 100644
--- a/lisp/international/fontset.el
+++ b/lisp/international/fontset.el
@@ -772,7 +772,8 @@ setup-default-fontset
   (dolist (math-subgroup '((#x1D400 #x1D433 mathematical-bold)
 			   (#x1D434 #x1D467 mathematical-italic)
 			   (#x1D468 #x1D49B mathematical-bold-italic)
-			   (#x1D49C #x1D4CF mathematical-script)
+			   (#x1D49C #x1D4B5 mathematical-script)
+			   (#x1D4B6 #x1D4CF mathematical-script)
 			   (#x1D4D0 #x1D503 mathematical-bold-script)
 			   (#x1D504 #x1D537 mathematical-fraktur)
 			   (#x1D538 #x1D56B mathematical-double-struck)

[-- Attachment #3: Type: text/plain, Size: 2026 bytes --]


script-representative-chars would contain:

> (mathematical-script . [119964 119990])

I meant that suggestion as a band-aid fix if we deem that Emacs should
guard against these "incomplete" fonts.

Not sure it makes sense as-is though, since it yields a vector; judging
by script-representative-chars's docstring (and a cursory glance at
ftfont_list and font_match_p), a list would be more appropriate, to make
all codepoints mandatory?

>>     1. (push #x1D4CF (alist-get 'mathematical-script
>>                                 script-representative-chars))
>>     2. (set-fontset-font
>>         t 'mathematical "DejaVu Math TeX Gyre" nil 'prepend)
>>     3. Uninstall KpMath.
>
> I'd suggest to do 3 regardless.  A font that supports only part of a
> block is a bad font to have.

(Right; AFAICT KpMath got pulled in when I installed TeXLive.  I'm not
entirely sure why the font was designed this way; I couldn't find a
rationale in the documentation hosted on CTAN[2])

> Emacs should probably do the equivalent of 1

OK; I'll try to follow-up with a bug report and a patch SomeDay™ then.


Again, thanks for your explanations.


[1]
$ fc-list ':charset=1d4c1'
⇒ a bunch of fonts, including DejaVu Math TeX Gyre but *not* KpMath

$ fc-match -f '%{charset}\n' 'KpMath'
… 1d456-1d49c 1d49e-1d49f 1d4a2 1d4a5-1d4a6 1d4a9-1d4ac 1d4ae-1d4b5
1d4d0-1d4e9 …
⇒ includes U+1D49C–U+1D4B5 (minus nonexistent codepoints) but not
  U+1D4B6–U+1D4CF

$ fc-match -f '%{charset}\n' 'DejaVu Math TeX Gyre'
… 1d456-1d49c 1d49e-1d49f 1d4a2 1d4a5-1d4a6 1d4a9-1d4ac 1d4ae-1d4b9
1d4bb 1d4bd-1d4c3 1d4c5-1d505…
⇒ includes U+1D49C–U+1D4B5 *and* U+1D4B6–U+1D4CF (minus nonexistent
  codepoints)

[2]
https://ctan.org/pkg/kpfonts
https://ctan.org/pkg/kpfonts-otf

The "List of glyphs" document from this second page, § 15.1.8, features
a table that clearly shows that KpMath (column "K") does not contain the
lowercase glyphs, but I fail to find a design rationale.

Re: script-representative-chars vs incomplete fonts

[Eli Zaretskii]

> From: Kévin Le Gouguec <kevin.legouguec@gmail.com>
> Cc: help-gnu-emacs@gnu.org
> Date: Sun, 12 Sep 2021 19:08:35 +0200
> 
> > It's not Emacs that stumbles on that font, it's your font backend
> > (probably fontconfig?).  Emacs just verifies that the candidate fonts
> > returned by the backend satisfy the requirements, and uses the first
> > font that does.  So this might mean a bug report for fontconfig and/or
> > the font designers is in order.
> 
> Right; "as luck would have it", fontconfig returns KpMath rather than
> DejaVu Math TeX Gyre; that's because Emacs only asks for U+1D49C, if I'm
> reading ftfont.c's use of Vscript_representative_chars correctly?

Emacs asks for some characters, but fontconfig should be smart enough
to return only fonts that support the Unicode block(s) of those
characters well enough.

> > What would be the purpose of breaking that range?
> > script-representative-chars can specify multiple characters per
> > script, see its doc string.
> 
> Right now script-representative-chars contains this:
> 
> > (mathematical-script 119964)
> 
> With this patch:
> 
> diff --git a/lisp/international/fontset.el b/lisp/international/fontset.el
> index 3deaff9677..461c8f830a 100644
> --- a/lisp/international/fontset.el
> +++ b/lisp/international/fontset.el
> @@ -772,7 +772,8 @@ setup-default-fontset
>    (dolist (math-subgroup '((#x1D400 #x1D433 mathematical-bold)
>  			   (#x1D434 #x1D467 mathematical-italic)
>  			   (#x1D468 #x1D49B mathematical-bold-italic)
> -			   (#x1D49C #x1D4CF mathematical-script)
> +			   (#x1D49C #x1D4B5 mathematical-script)
> +			   (#x1D4B6 #x1D4CF mathematical-script)
>  			   (#x1D4D0 #x1D503 mathematical-bold-script)
>  			   (#x1D504 #x1D537 mathematical-fraktur)
>  			   (#x1D538 #x1D56B mathematical-double-struck)
> 
> script-representative-chars would contain:
> 
> > (mathematical-script . [119964 119990])
> 
> I meant that suggestion as a band-aid fix if we deem that Emacs should
> guard against these "incomplete" fonts.

I think it is easier to request both upper-case and lower-case
letters; since the lower-case letters are at fixed offsets from their
upper-case variants, we can compute the codepoints without changing
the original database.

> Not sure it makes sense as-is though, since it yields a vector; judging
> by script-representative-chars's docstring (and a cursory glance at
> ftfont_list and font_match_p), a list would be more appropriate, to make
> all codepoints mandatory?

Why do you say that a list would be more appropriate?  The doc string
of script-representative-chars says:

   CHARS is a list or a vector of characters.

So both lists and vectors are possible and supported.

> > Emacs should probably do the equivalent of 1
> 
> OK; I'll try to follow-up with a bug report and a patch SomeDay™ then.

Thanks.

Re: script-representative-chars vs incomplete fonts

[Kévin Le Gouguec]

Eli Zaretskii <eliz@gnu.org> writes:

> I think it is easier to request both upper-case and lower-case
> letters; since the lower-case letters are at fixed offsets from their
> upper-case variants, we can compute the codepoints without changing
> the original database.

Noted.

>> Not sure it makes sense as-is though, since it yields a vector; judging
>> by script-representative-chars's docstring (and a cursory glance at
>> ftfont_list and font_match_p), a list would be more appropriate, to make
>> all codepoints mandatory?
>
> Why do you say that a list would be more appropriate?  The doc string
> of script-representative-chars says:
>
>    CHARS is a list or a vector of characters.
>
> So both lists and vectors are possible and supported.

The docstring also says this:

    If it is a list, all characters in the list are necessary for
    supporting SCRIPT.
    If it is a vector, one of the characters in the vector is necessary.

That led me to think that we'd want a list, but now that I've looked at
ftfont.c and font.c more closely, I get the impression that this
distinction only applies to the verification step in font_match_p?
Hence it would indeed not matter whether we used a vector or a list for
the purposes of querying fontconfig.

(I hope I'm not misreading the code; apologies if so.  I'm still not
entirely sure I understand why ftfont_spec_pattern seems to only handle
the list case while ftfont_list only handles the vector case, but I'm
sure I'll figure it out eventually after some more scowling)

Again, thanks for your advice.

Re: script-representative-chars vs incomplete fonts

[Eli Zaretskii]

> From: Kévin Le Gouguec <kevin.legouguec@gmail.com>
> Cc: help-gnu-emacs@gnu.org
> Date: Mon, 13 Sep 2021 00:21:30 +0200
> 
> >    CHARS is a list or a vector of characters.
> >
> > So both lists and vectors are possible and supported.
> 
> The docstring also says this:
> 
>     If it is a list, all characters in the list are necessary for
>     supporting SCRIPT.
>     If it is a vector, one of the characters in the vector is necessary.

So you want to require both upper- and lower-case variants?

> That led me to think that we'd want a list, but now that I've looked at
> ftfont.c and font.c more closely, I get the impression that this
> distinction only applies to the verification step in font_match_p?

Basically, yes.

> Hence it would indeed not matter whether we used a vector or a list for
> the purposes of querying fontconfig.

No, but if we require both upper- and lower-case variants, we will
reject fonts such as KpMath.

> (I hope I'm not misreading the code; apologies if so.  I'm still not
> entirely sure I understand why ftfont_spec_pattern seems to only handle
> the list case while ftfont_list only handles the vector case, but I'm
> sure I'll figure it out eventually after some more scowling)

ftfont_list calls ftfont_spec_pattern, so both variants are handled.
My reading of that code is that ftfont.c uses
script-representative-chars as fallback for the cases when the fontset
doesn't specify a script or a charset for a font.  ftfont.c then
creates a fake "charset" on the fly out of the representative
characters.

Re: script-representative-chars vs incomplete fonts

[Kévin Le Gouguec]

[-- Attachment #1: Type: text/plain, Size: 429 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> The docstring also says this:
>> 
>>     If it is a list, all characters in the list are necessary for
>>     supporting SCRIPT.
>>     If it is a vector, one of the characters in the vector is necessary.
>
> So you want to require both upper- and lower-case variants?

As a mitigation against fonts like KpMath, yes.  See attached debugging
code, and subsequent patch/screenshot pairs.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: test-fontset.el --]
[-- Type: text/x-emacs-lisp, Size: 1772 bytes --]

(with-current-buffer (get-buffer-create "foo")
  (pcase-dolist (`(,start ,end ,group) '((#x1D400 #x1D433 mathematical-bold)
			                 (#x1D434 #x1D467 mathematical-italic)
			                 (#x1D468 #x1D49B mathematical-bold-italic)
			                 (#x1D49C #x1D4CF mathematical-script)
			                 (#x1D4D0 #x1D503 mathematical-bold-script)
			                 (#x1D504 #x1D537 mathematical-fraktur)
			                 (#x1D538 #x1D56B mathematical-double-struck)
			                 (#x1D56C #x1D59F mathematical-bold-fraktur)
			                 (#x1D5A0 #x1D5D3 mathematical-sans-serif)
			                 (#x1D5D4 #x1D607 mathematical-sans-serif-bold)
			                 (#x1D608 #x1D63B mathematical-sans-serif-italic)
			                 (#x1D63C #x1D66F mathematical-sans-serif-bold-italic)
			                 (#x1D670 #x1D6A3 mathematical-monospace)
			                 (#x1D6A4 #x1D6A5 mathematical-italic)
			                 (#x1D6A8 #x1D6E1 mathematical-bold)
			                 (#x1D6E2 #x1D71B mathematical-italic)
			                 (#x1D71C #x1D755 mathematical-bold-italic)
			                 (#x1D756 #x1D78F mathematical-sans-serif-bold)
			                 (#x1D790 #x1D7C9 mathematical-sans-serif-bold-italic)
			                 (#x1D7CA #x1D7D7 mathematical-bold)
			                 (#x1D7D8 #x1D7E1 mathematical-double-struck)
			                 (#x1D7E2 #x1D7EB mathematical-sans-serif)
			                 (#x1D7EC #x1D7F5 mathematical-sans-serif-bold)
			                 (#x1D7F6 #x1D7FF mathematical-monospace)))
    (insert (format "%s: %s-%s\n" group (char-to-string start) (char-to-string end))))
  (switch-to-buffer "foo")
  (split-window-horizontally)
  (describe-variable 'script-representative-chars))

[-- Attachment #3: Type: text/plain, Size: 23 bytes --]


Emacs 28, unpatched:


[-- Attachment #4: 0-unpatched.png --]
[-- Type: image/png, Size: 217408 bytes --]

[-- Attachment #5: Type: text/plain, Size: 43 bytes --]


"Vector" patch, with even worse results:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #6: fontset-vector.patch --]
[-- Type: text/x-patch, Size: 995 bytes --]

diff --git a/lisp/international/fontset.el b/lisp/international/fontset.el
index 3deaff9677..31950dae85 100644
--- a/lisp/international/fontset.el
+++ b/lisp/international/fontset.el
@@ -795,10 +795,11 @@ setup-default-fontset
 			   (#x1D7F6 #x1D7FF mathematical-monospace)))
     (let ((slot (assq (nth 2 math-subgroup) script-representative-chars)))
       (if slot
-	  (if (vectorp (cdr slot))
-	      (setcdr slot (vconcat (cdr slot) (vector (car math-subgroup))))
-	    (setcdr slot (vector (cadr slot) (car math-subgroup))))
-	(setq slot (list (nth 2 math-subgroup) (car math-subgroup)))
+          (setcdr slot (vconcat (cdr slot) (vector (car math-subgroup)
+                                                   (cadr math-subgroup))))
+	(setq slot (list (nth 2 math-subgroup)
+                         (vector (car math-subgroup)
+                                 (cadr math-subgroup))))
 	(nconc script-representative-chars (list slot))))
     (set-fontset-font
      "fontset-default"

[-- Attachment #7: 1-vector.png --]
[-- Type: image/png, Size: 189325 bytes --]

[-- Attachment #8: Type: text/plain, Size: 43 bytes --]


"List" patch, with satisfactory results:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #9: fontset-list.patch --]
[-- Type: text/x-patch, Size: 993 bytes --]

diff --git a/lisp/international/fontset.el b/lisp/international/fontset.el
index 3deaff9677..0236cd790a 100644
--- a/lisp/international/fontset.el
+++ b/lisp/international/fontset.el
@@ -795,10 +795,12 @@ setup-default-fontset
 			   (#x1D7F6 #x1D7FF mathematical-monospace)))
     (let ((slot (assq (nth 2 math-subgroup) script-representative-chars)))
       (if slot
-	  (if (vectorp (cdr slot))
-	      (setcdr slot (vconcat (cdr slot) (vector (car math-subgroup))))
-	    (setcdr slot (vector (cadr slot) (car math-subgroup))))
-	(setq slot (list (nth 2 math-subgroup) (car math-subgroup)))
+          (setcdr slot (append (list (car math-subgroup)
+                                     (cadr math-subgroup))
+                               (cdr slot)))
+	(setq slot (list (nth 2 math-subgroup)
+                         (car math-subgroup)
+                         (cadr math-subgroup)))
 	(nconc script-representative-chars (list slot))))
     (set-fontset-font
      "fontset-default"

[-- Attachment #10: 2-list.png --]
[-- Type: image/png, Size: 224984 bytes --]

[-- Attachment #11: Type: text/plain, Size: 88 bytes --]


Alternative "list" patch, with fewer mandatory codepoints and similarly
good results:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #12: fontset-26.patch --]
[-- Type: text/x-patch, Size: 2144 bytes --]

diff --git a/lisp/international/fontset.el b/lisp/international/fontset.el
index 3deaff9677..533deda61e 100644
--- a/lisp/international/fontset.el
+++ b/lisp/international/fontset.el
@@ -769,7 +769,9 @@ setup-default-fontset
 		      nil 'append))
 
   ;; Special settings for `MATHEMATICAL (U+1D400..U+1D7FF)'.
-  (dolist (math-subgroup '((#x1D400 #x1D433 mathematical-bold)
+  (dolist (math-subgroup '(
+                           ;; First pass: A-z range.
+                           (#x1D400 #x1D433 mathematical-bold)
 			   (#x1D434 #x1D467 mathematical-italic)
 			   (#x1D468 #x1D49B mathematical-bold-italic)
 			   (#x1D49C #x1D4CF mathematical-script)
@@ -782,6 +784,7 @@ setup-default-fontset
 			   (#x1D608 #x1D63B mathematical-sans-serif-italic)
 			   (#x1D63C #x1D66F mathematical-sans-serif-bold-italic)
 			   (#x1D670 #x1D6A3 mathematical-monospace)
+                           ;; Additional passes: dotless, greek, digits, etc.
 			   (#x1D6A4 #x1D6A5 mathematical-italic)
 			   (#x1D6A8 #x1D6E1 mathematical-bold)
 			   (#x1D6E2 #x1D71B mathematical-italic)
@@ -792,13 +795,16 @@ setup-default-fontset
 			   (#x1D7D8 #x1D7E1 mathematical-double-struck)
 			   (#x1D7E2 #x1D7EB mathematical-sans-serif)
 			   (#x1D7EC #x1D7F5 mathematical-sans-serif-bold)
-			   (#x1D7F6 #x1D7FF mathematical-monospace)))
+			   (#x1D7F6 #x1D7FF mathematical-monospace)
+                           ))
     (let ((slot (assq (nth 2 math-subgroup) script-representative-chars)))
       (if slot
-	  (if (vectorp (cdr slot))
-	      (setcdr slot (vconcat (cdr slot) (vector (car math-subgroup))))
-	    (setcdr slot (vector (cadr slot) (car math-subgroup))))
-	(setq slot (list (nth 2 math-subgroup) (car math-subgroup)))
+          (push (car math-subgroup) (cdr slot))
+        ;; First pass: add both CAPITAL A and SMALL A to guard against
+        ;; fonts that only cover capital codepoints.
+	(setq slot (list (nth 2 math-subgroup)
+                         (car math-subgroup)
+                         (+ 26 (car math-subgroup))))
 	(nconc script-representative-chars (list slot))))
     (set-fontset-font
      "fontset-default"

[-- Attachment #13: 3-list26.png --]
[-- Type: image/png, Size: 230318 bytes --]

[-- Attachment #14: Type: text/plain, Size: 112 bytes --]


Should I submit (a variation of) one of these patches to bug-gnu-emacs?
Or should I look into other solutions?

Re: script-representative-chars vs incomplete fonts

[Eli Zaretskii]

> From: Kévin Le Gouguec <kevin.legouguec@gmail.com>
> Cc: help-gnu-emacs@gnu.org
> Date: Tue, 14 Sep 2021 00:46:24 +0200
> 
> Should I submit (a variation of) one of these patches to bug-gnu-emacs?

Thanks, I think the 2nd one is the best, as it's simple and does the
job.  Yes, please submit to the bug tracker.