bug#45184: 26.1; Private Unicode plains 15 and 16 seem not supported

bug#45184: 26.1; Private Unicode plains 15 and 16 seem not supported

[Janusz S. Bień]

I intend to use plains 15 and/or 16 for the historical variant of Polish
letters. An appropriate font is understand preparation, but not yet
available publicty.

When the font was installed in ./loca/share/fonts, a QT program found it
and displayed the characters correctly. Emacs, to the contrary, for

character: 󰀀 (displayed as 󰀀) (codepoint 983040, #o3600000, #xf0000)

reports "no font available".

Best regards

Janusz


In GNU Emacs 26.1 (build 2, x86_64-pc-linux-gnu, GTK+ Version 3.24.5)
 of 2019-09-23, modified by Debian built on x86-grnet-01
Windowing system distributor 'The X.Org Foundation', version 11.0.12004000
System Description:	Debian GNU/Linux 10 (buster)


-- 
             ,   
Janusz S. Bien
emeryt (emeritus)
https://sites.google.com/view/jsbien

bug#45184: 26.1; Private Unicode plains 15 and 16 seem not supported

[Eli Zaretskii]

severity 45184 wishlist
thanks

> From: jsbien@mimuw.edu.pl (Janusz S. Bień)
> Date: Fri, 11 Dec 2020 18:55:51 +0100
> 
> I intend to use plains 15 and/or 16 for the historical variant of Polish
> letters. An appropriate font is understand preparation, but not yet
> available publicty.
> 
> When the font was installed in ./loca/share/fonts, a QT program found it
> and displayed the characters correctly. Emacs, to the contrary, for
> 
> character: 󰀀 (displayed as 󰀀) (codepoint 983040, #o3600000, #xf0000)
> 
> reports "no font available".

That's because Emacs has no idea that you are using SPUA planes for
Polish.  You need to tell Emacs about that by augmenting your fontset.
(And even after you do that, there could be problems, because Emacs
doesn't treat PUA ranges of Unicode codepoints as characters.)

bug#45184: 26.1; Private Unicode plains 15 and 16 seem not supported

[Janusz S. Bień]

On Fri, Dec 11 2020 at 22:10 +02, Eli Zaretskii wrote:
> severity 45184 wishlist

This is a bug report, not a feature request.

> thanks
>
>> From: jsbien@mimuw.edu.pl (Janusz S. Bień)
>> Date: Fri, 11 Dec 2020 18:55:51 +0100
>> 
>> I intend to use plains 15 and/or 16 for the historical variant of Polish
>> letters. An appropriate font is understand preparation, but not yet
>> available publicty.
>> 
>> When the font was installed in ./loca/share/fonts, a QT program found it
>> and displayed the characters correctly. Emacs, to the contrary, for
>> 
>> character: 󰀀 (displayed as 󰀀) (codepoint 983040, #o3600000, #xf0000)
>> 
>> reports "no font available".
>
> That's because Emacs has no idea that you are using SPUA planes for
> Polish.

The fact that I want to use the characters for Polish is irrelevant.

> You need to tell Emacs about that by augmenting your fontset.

The PUA from the Basic Multulingual Plane are used by me for years and
do not require doing anything with the fontset. For example, LATIN SMALL
LETTER LONG S WITH FLOURISH (coming from the Medieval Unicode Font
Initiative) is displayed without problem and the appropriate font is
found automatically:

            character:  (displayed as ) (codepoint 59575, #o164267, #xe8b7)
    xft:-psbk-Junicode-normal-normal-normal-*-15-*-*-*-*-0-iso10646-1 (#x94C)

The same should happen for the characters from plains 15 and 16. However
now even the explicit selection of the appropriate font doesn't help.

> (And even after you do that, there could be problems, because Emacs
> doesn't treat PUA ranges of Unicode codepoints as characters.)

Really? What do you exactly mean? My experience contradicts this
statement.

I think the Emacs code just for some reason limits the range of the
Unicode characters, and the limitation should be removed.

Regards

Janusz

-- 
             ,   
Janusz S. Bien
emeryt (emeritus)
https://sites.google.com/view/jsbien

bug#45184: 26.1; Private Unicode plains 15 and 16 seem not supported

[Eli Zaretskii]

> From: jsbien@mimuw.edu.pl (Janusz S. Bień)
> Cc: 45184@debbugs.gnu.org
> Date: Sat, 12 Dec 2020 07:19:15 +0100
> 
> On Fri, Dec 11 2020 at 22:10 +02, Eli Zaretskii wrote:
> > severity 45184 wishlist
> 
> This is a bug report, not a feature request.

It is a report about a missing functionality, thus a feature request.

> > That's because Emacs has no idea that you are using SPUA planes for
> > Polish.
> 
> The fact that I want to use the characters for Polish is irrelevant.

Not the way Emacs looks up fonts installed on the system.  It uses the
character's script and then a database of representative script
characters to find appropriate candidate fonts.

> > You need to tell Emacs about that by augmenting your fontset.
> 
> The PUA from the Basic Multulingual Plane are used by me for years and
> do not require doing anything with the fontset. For example, LATIN SMALL
> LETTER LONG S WITH FLOURISH (coming from the Medieval Unicode Font
> Initiative) is displayed without problem and the appropriate font is
> found automatically:
> 
>             character:  (displayed as ) (codepoint 59575, #o164267, #xe8b7)
>     xft:-psbk-Junicode-normal-normal-normal-*-15-*-*-*-*-0-iso10646-1 (#x94C)

AFAIK, this happens by sheer luck.

Of course, my expertise in this part of Emacs is rather limited, so I
might be missing something.  Corrections with explanations are
welcome.

> > (And even after you do that, there could be problems, because Emacs
> > doesn't treat PUA ranges of Unicode codepoints as characters.)
> 
> Really? What do you exactly mean? My experience contradicts this
> statement.

Please look at the various settings of character-related databases in
Emacs: they basically deny PUA codepoints any reasonable character
properties.  IOW, Emacs doesn't know anything about these codepoints,
in terms of the character attributes and properties it needs for
proper support.  Font selection is just the tip of the iceberg.

The feature we miss is some facility to tell Emacs the semantics of a
range of PUA codepoints, so that the rest of the code will support
them properly and correctly.

> I think the Emacs code just for some reason limits the range of the
> Unicode characters, and the limitation should be removed.

Any evidence that such a limitation exists and/or where does it
happen?

bug#45184: 26.1; Private Unicode plains 15 and 16 seem not supported

[Janusz S. Bień]

On Sat, Dec 12 2020 at  9:45 +02, Eli Zaretskii wrote:
>> From: jsbien@mimuw.edu.pl (Janusz S. Bień)
>> Cc: 45184@debbugs.gnu.org
>> Date: Sat, 12 Dec 2020 07:19:15 +0100
>> 
>> On Fri, Dec 11 2020 at 22:10 +02, Eli Zaretskii wrote:
>> > severity 45184 wishlist
>> 
>> This is a bug report, not a feature request.
>
> It is a report about a missing functionality, thus a feature request.
>
>> > That's because Emacs has no idea that you are using SPUA planes for
>> > Polish.
>> 
>> The fact that I want to use the characters for Polish is irrelevant.
>
> Not the way Emacs looks up fonts installed on the system.  It uses the
> character's script and then a database of representative script
> characters to find appropriate candidate fonts.

Where I can find more details about the process?

Regards

Janusz

-- 
             ,   
Janusz S. Bien
emeryt (emeritus)
https://sites.google.com/view/jsbien

bug#45184: 26.1; Private Unicode plains 15 and 16 seem not supported

[Janusz S. Bień]

On Sat, Dec 12 2020 at  9:45 +02, Eli Zaretskii wrote:

[...]

>> I think the Emacs code just for some reason limits the range of the
>> Unicode characters, and the limitation should be removed.
>
> Any evidence that such a limitation exists and/or where does it
> happen?

The chacters are not displayed despite the fact that the appropriate
font is selected explicitely.

Janusz

-- 
             ,   
Janusz S. Bien
emeryt (emeritus)
https://sites.google.com/view/jsbien

bug#45184: 26.1; Private Unicode plains 15 and 16 seem not supported

[Andreas Schwab]

On Dez 11 2020, Janusz S. Bień wrote:

> When the font was installed in ./loca/share/fonts, a QT program found it
> and displayed the characters correctly. Emacs, to the contrary, for
>
> character: 󰀀 (displayed as 󰀀) (codepoint 983040, #o3600000, #xf0000)
>
> reports "no font available".

Did you restart Emacs after installing the font?

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

bug#45184: 26.1; Private Unicode plains 15 and 16 seem not supported

[Janusz S. Bień]

On Sat, Dec 12 2020 at 10:01 +01, Andreas Schwab wrote:
> On Dez 11 2020, Janusz S. Bień wrote:
>
>> When the font was installed in ./loca/share/fonts, a QT program found it
>> and displayed the characters correctly. Emacs, to the contrary, for
>>
>> character: 󰀀 (displayed as 󰀀) (codepoint 983040, #o3600000, #xf0000)
>>
>> reports "no font available".
>
> Did you restart Emacs after installing the font?

Thank you very much for your interest in the problem. 

Actually the system was even restarted. The first test was done
yesterday but it persist today after powering the computer.

Please have a look at the recent updates

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=45184

I would appreciate very much your comments to Eliz Zaretski's claims.

Regards

Janusz

-- 
             ,   
Janusz S. Bien
emeryt (emeritus)
https://sites.google.com/view/jsbien

bug#45184: 26.1; Private Unicode plains 15 and 16 seem not supported

[Eli Zaretskii]

> From: jsbien@mimuw.edu.pl (Janusz S. Bień)
> Cc: 45184@debbugs.gnu.org
> Date: Sat, 12 Dec 2020 09:43:53 +0100
> 
> >> > That's because Emacs has no idea that you are using SPUA planes for
> >> > Polish.
> >> 
> >> The fact that I want to use the characters for Polish is irrelevant.
> >
> > Not the way Emacs looks up fonts installed on the system.  It uses the
> > character's script and then a database of representative script
> > characters to find appropriate candidate fonts.
> 
> Where I can find more details about the process?

Unfortunately, mainly in the sources: font.c, fontset.c, and
fontset.el are the main places to look.  This area is notoriously
under-documented in Emacs.

bug#45184: 26.1; Private Unicode plains 15 and 16 seem not supported

[Eli Zaretskii]

> From: jsbien@mimuw.edu.pl (Janusz S. Bień)
> Cc: 45184@debbugs.gnu.org
> Date: Sat, 12 Dec 2020 09:49:09 +0100
> 
> On Sat, Dec 12 2020 at  9:45 +02, Eli Zaretskii wrote:
> 
> [...]
> 
> >> I think the Emacs code just for some reason limits the range of the
> >> Unicode characters, and the limitation should be removed.
> >
> > Any evidence that such a limitation exists and/or where does it
> > happen?
> 
> The chacters are not displayed despite the fact that the appropriate
> font is selected explicitely.

I meant evidence that the root cause of this is some limitation on the
range of Unicode characters.  There's no dispute about the facts, but
I'm not aware of any limitation of the kind that you suggested.

Also, please describe in detail how you selected the font in question.
I don't think you told that in your original report.

bug#45184: 26.1; Private Unicode plains 15 and 16 seem not supported

[Janusz S. Bień]

On Sat, Dec 12 2020 at 12:06 +02, Eli Zaretskii wrote:
>> From: jsbien@mimuw.edu.pl (Janusz S. Bień)

[...]

>> > Any evidence that such a limitation exists and/or where does it
>> > happen?
>> 
>> The chacters are not displayed despite the fact that the appropriate
>> font is selected explicitely.
>
> I meant evidence that the root cause of this is some limitation on the
> range of Unicode characters.  There's no dispute about the facts, but
> I'm not aware of any limitation of the kind that you suggested.

You are right, perhaps I jumped to a wrong conclusion, but it seemed to
me the simplest explanatation. Perhaps my bug report (I still think it
is a bug) should be renamed to "Emacs doesn't find font for some
characters".

> Also, please describe in detail how you selected the font in question.
> I don't think you told that in your original report.

I didn't told because I expected the font to be found automatically.

In an additional experiment I just used `S-down-mouse-1’ and selected
the font from the list.

Please note that vim (installed especially for testing) finds the font
automatically and displays the characters correctly.

Janusz

-- 
             ,   
Janusz S. Bien
emeryt (emeritus)
https://sites.google.com/view/jsbien

bug#45184: 26.1; Private Unicode plains 15 and 16 seem not supported

[Eli Zaretskii]

> From: jsbien@mimuw.edu.pl (Janusz S. Bień)
> Cc: 45184@debbugs.gnu.org
> Date: Sat, 12 Dec 2020 11:29:07 +0100
> 
> > Also, please describe in detail how you selected the font in question.
> > I don't think you told that in your original report.
> 
> I didn't told because I expected the font to be found automatically.
> 
> In an additional experiment I just used `S-down-mouse-1’ and selected
> the font from the list.

And what happens then? are any characters displayed correctly, or all
of them are displayed as hex codes?

> Please note that vim (installed especially for testing) finds the font
> automatically and displays the characters correctly.

I don't know how vim selects the fonts, but I'm sure it's very
different from what Emacs does.  We need to understand what prevents
Emacs from using this font as you expected.

Anyway, I think it's time to present more details, if you please:

  . does this happen in "emacs -Q"?
  . did you customize your fontsets, and if so, can you show those
    customizations?
  . which codepoints are used for the problematic characters, and
    which characters are those? do they have codepoints in Unicode?
  . can you make the font in question available for download, so we
    could try debugging this in Emacs?
  . assuming the font is installed, can you show a minimal recipe for
    reproducing the problem in "emacs -Q"?

TIA

bug#45184: 26.1; Private Unicode plains 15 and 16 seem not supported

[Janusz S. Bień]

On Sat, Dec 12 2020 at 11:29 +01, Janusz S. Bień wrote:
> On Sat, Dec 12 2020 at 12:06 +02, Eli Zaretskii wrote:
>>> From: jsbien@mimuw.edu.pl (Janusz S. Bień)

[...]

> In an additional experiment I just used `S-down-mouse-1’ and selected
> the font from the list.

When I make the font the default one in the command line

emacs -fn "JuniusX"

the characters are displayed correctly.

So I suggest to close this bug. 

There are definitely some problems with font selection, but I have to
investigate them more carefully.

Best regards

Janusz

-- 
             ,   
Janusz S. Bien
emeryt (emeritus)
https://sites.google.com/view/jsbien

bug#45184: 26.1; Private Unicode plains 15 and 16 seem not supported

[Eli Zaretskii]

> From: jsbien@mimuw.edu.pl (Janusz S. Bień)
> Cc: 45184@debbugs.gnu.org
> Date: Sat, 12 Dec 2020 19:30:21 +0100
> 
> When I make the font the default one in the command line
> 
> emacs -fn "JuniusX"
> 
> the characters are displayed correctly.

This probably means that if you customize fontset-default to use this
font for the range of codepoints you want it to be used, the problem
will be resolved even without specifying the font as the default one.

> So I suggest to close this bug. 

OK, done.

> There are definitely some problems with font selection, but I have to
> investigate them more carefully.

When you have reached some conclusions, please reopen this bug or file
a new one with the details.

Thanks.