bug#31315: wrong font encoding for fallback font

[Eli Zaretskii <eliz@gnu.org>]

> Date: Mon, 30 Apr 2018 09:21:06 +0200 (CEST)
> From: Werner LEMBERG <wl@gnu.org>

(Adding Handa-san to the discussion in the hope that he might have
some comments.)

> The attached image shows that some CJK characters are displayed
> incorrectly.  For the used outline font Emacs reports
> 
>   xft:-PfEd-AR PL UKai TW MBE-normal-normal-normal-*-18-*-*-*-*-0-iso10646-1
> 
> One character (the highlighted one) is missing in this font, and Emacs
> uses a different font as a fallback:
> 
>   x:-misc-droid sans fallback-medium-r-normal--18-130-100-100-p-179-gb18030.2000-0
> 
> Note that the different font backend seems to produce the ugly
> rendering; the font in question is the outline font
> `DroidSansFallbackFull.ttf'.

That's why it is fallback, I guess... ;-)

And I think you might be mistaken in your interpretation of what
"gb18030.2000" in the font name means: I think it's the font registry,
not its encoding.  How sure are you that the encoding of this font is
indeed gb18030.2000?  (I'm not an expert on this stuff.)

> The problem now is that the encoding of the fallback font is not
> respected.  In the image, the highlighted character is U+83EF, but
> Emacs incorrectly displays U+51BF instead.
> 
> The GB 18030 bytes to represent U+51BF are \x83\xEF; this clearly
> shows that Emacs lacks an iconv call (or an equivalent to that);
> instead, it seems to simply feed the Unicode value to the font
> backend.

Tz-tz-tz, how can you even suggest something like that about Emacs ;-)

If you look in xfont_encode_char, you will see that it does encode the
character before handing it to the font-drawing function.  But I see
that font-encoding-alist has this to say about gb18030:

 ("gb18030" unicode)

Does replacing that with something like this:

 ("gb18030" (gb18030 . unicode))

solve the problem?

What we put in font-encoding-alist now was a deliberate change in Jan
2008, in response to a bug report; see

  http://lists.gnu.org/archive/html/emacs-devel/2008-01/msg00754.html

If fonts like this one need to have characters encoded by gb18030,
then I think we need to change what the value says.  But this area in
Emacs is under-documented, so I'm not sure I've got it right, in
particular what is the effect of ENCODING and REPERTORY in this
context.  For most font back-ends, ENCODING is ignored, because the
back-end is capable to encode the character we hand to it.  But the
xfont back-end indeed uses Emacs's encoding functions to do that
externally to the corresponding X APIs.  Which might explain why this
problem, if indeed we fail to specify the correct encoding for this
charset, was never reported till now: xfont is rarely if ever used.

> It's a completely different question why on my system Emacs uses a
> font encoded in GB 18030 as a fallback font.  It's probably related to
> the fact that I use `mew' as my e-mail program, manually extended to
> cover GB 18030.  Unfortunately, I wasn't able yet to trigger the issue
> with `emacs -Q' (which by default uses iso10646 for the fallback
> font).

Well, we cannot try helping you to unlock this unless you tell how you
"manually extended" Emacs.  In general, the way to request that Emacs
uses fonts you like with certain characters or charsets is by
customizing your fontsets.  I cannot say more without hearing the
details.

> On the other hand, as soon as the problem happens, it happens
> with any buffer containing CJK characters not displayable with the
> current font, so it seems a genuine Emacs core bug.

What "problem" do you allude to here?  The first (seemingly incorrect
encoding) or the second (fallback to this particular font)?