Face issue -- possible bug

Face issue -- possible bug

[Joost Kremers]

[-- Attachment #1: Type: text/plain, Size: 770 bytes --]

Hi all,

I'm running into an issue with a particular font. I'm not sure if 
this is a problem with Emacs or with the font, so before reporting 
an actual bug, I thought I'd ask here first.

The font I'm having trouble with is Linux Libertine O, which is a 
variable-width font, so it only shows when `variable-pitch-mode` 
is enabled.

The issue occurs with several diacritics of the International 
Phonetic Alphabet[1], which are combining characters. One 
character causing the issue is ̩ COMBINING VERTICAL LINE BELOW 
0x0329. When this character is combined with e.g., n, the result 
is displayed as a (small) capital n with the diacritic, not as an 
lower case n.

The screen show below shows the problem (it's the final letter in 
the word):


[-- Attachment #2: Screenshot from 2018-07-16 22-26-15.png --]
[-- Type: image/png, Size: 66394 bytes --]

[-- Attachment #3: Type: text/plain, Size: 279 bytes --]


This screen shot was taken with `emacs -Q` and executing 
`(set-face-attribute 'variable-pitch nil :height 1.4 :family 
"Linux Libertine O")` at an `IELM` prompt and `M-x 
variable-pitch-mode` in the buffer. To see what the word *should* 
look like, consider this screen shot:


[-- Attachment #4: Screenshot from 2018-07-16 22-27-37.png --]
[-- Type: image/png, Size: 56779 bytes --]

[-- Attachment #5: Type: text/plain, Size: 394 bytes --]


This is also `emacs -Q`, with `(set-face-attribute 'default nil 
:height 110 :foundry "unknown" :family "DejaVu Sans Mono")` and 
`variable-pitch-mode` disabled.

The fact that Emacs displays the word correctly with DejaVu Sans 
Mono seems to suggest that it's not an Emacs problem but rather a 
font problem, but note that the same diacritic with the same font 
displays correctly in a pdf:


[-- Attachment #6: Screenshot from 2018-07-16 22-42-24.png --]
[-- Type: image/png, Size: 28770 bytes --]

[-- Attachment #7: Type: text/plain, Size: 327 bytes --]


(That's a screen shot of the pdf resulting from the LaTeX file in 
the first two screen shots, created with xelatex).

So should this be reported as an Emacs bug, or should I contact 
the font creator?

TIA

Joost



[1] 
<https://en.wikipedia.org/wiki/International_Phonetic_Alphabet>

-- 
Joost Kremers
Life has its moments

Re: Face issue -- possible bug

[Eli Zaretskii]

> From: Joost Kremers <joostkremers@fastmail.fm>
> Date: Mon, 16 Jul 2018 22:49:18 +0200
> 
> So should this be reported as an Emacs bug, or should I contact 
> the font creator?

What does Emacs say if you go to the character and type "C-u C-x =",
when the problematic font is used?

Re: Face issue -- possible bug

[Joost Kremers]

On Tue, Jul 17 2018, Eli Zaretskii wrote:
>> From: Joost Kremers <joostkremers@fastmail.fm>
>> Date: Mon, 16 Jul 2018 22:49:18 +0200
>> 
>> So should this be reported as an Emacs bug, or should I contact 
>> the font creator?
>
> What does Emacs say if you go to the character and type "C-u C-x 
> =",
> when the problematic font is used?

Well d'uh, I could have thought of that myself... Anyway, with 
Linux Libertine O (which displays the character incorrectly), `C-u 
C-x =` gives:

==============================

             position: 182 of 268 (68%), column: 5
            character: n (displayed as n) (codepoint 110, #o156, 
            #x6e)
    preferred charset: ascii (ASCII (ISO646 IRV))
code point in charset: 0x6E
               script: latin
               syntax: w 	which means: word
             category: .:Base, L:Left-to-right (strong), a:ASCII, 
             l:Latin, r:Roman
             to input: type "C-x 8 RET 6e" or "C-x 8 RET LATIN 
             SMALL LETTER N"
          buffer code: #x6E
            file code: #x6E (encoded by coding system utf-8-unix)
              display: composed to form "n̩" (see below)

Composed with the following character(s) "̩" using this font:
  xft:-PfEd-Linux Libertine 
  O-normal-normal-normal-*-21-*-*-*-*-0-iso10646-1
by these glyphs:
  [0 1 0 2420 13 0 13 10 1 nil]
  [0 1 809 745 0 -9 -6 -1 5 [-2 -1 0]]

Character code properties: customize what to show
  name: LATIN SMALL LETTER N
  general-category: Ll (Letter, Lowercase)
  decomposition: (110) ('n')

There are text properties here:
  fontified            t
  wrap-prefix          ""

==============================

And with DejaVu Sans Mono (displaying the character correctly):

==============================


             position: 182 of 268 (68%), column: 5
            character: n (displayed as n) (codepoint 110, #o156, 
            #x6e)
    preferred charset: ascii (ASCII (ISO646 IRV))
code point in charset: 0x6E
               script: latin
               syntax: w 	which means: word
             category: .:Base, L:Left-to-right (strong), a:ASCII, 
             l:Latin, r:Roman
             to input: type "C-x 8 RET 6e" or "C-x 8 RET LATIN 
             SMALL LETTER N"
          buffer code: #x6E
            file code: #x6E (encoded by coding system utf-8-unix)
              display: composed to form "n̩" (see below)

Composed with the following character(s) "̩" using this font:
  xft:-PfEd-DejaVu Sans 
  Mono-normal-normal-normal-*-15-*-*-*-m-0-iso10646-1
by these glyphs:
  [0 1 110 81 9 1 8 8 5 nil]
  [0 1 809 689 0 4 6 -1 5 [-10 -1 0]]

Character code properties: customize what to show
  name: LATIN SMALL LETTER N
  general-category: Ll (Letter, Lowercase)
  decomposition: (110) ('n')

There are text properties here:
  fontified            t
  wrap-prefix          ""

==============================

The only difference I notice is with the numbers following "by 
these glyphs", but I have no idea what those numbers mean.

TIA

Joost



-- 
Joost Kremers
Life has its moments

Re: Face issue -- possible bug

[Eli Zaretskii]

> From: Joost Kremers <joostkremers@fastmail.fm>
> Cc: emacs-devel@gnu.org
> Date: Tue, 17 Jul 2018 09:56:14 +0200
> 
> > What does Emacs say if you go to the character and type "C-u C-x 
> > =",
> > when the problematic font is used?
> 
> Well d'uh, I could have thought of that myself... Anyway, with 
> Linux Libertine O (which displays the character incorrectly), `C-u 
> C-x =` gives:
> 
> ==============================
> 
>              position: 182 of 268 (68%), column: 5
>             character: n (displayed as n) (codepoint 110, #o156, 
>             #x6e)
>     preferred charset: ascii (ASCII (ISO646 IRV))
> code point in charset: 0x6E
>                script: latin
>                syntax: w 	which means: word
>              category: .:Base, L:Left-to-right (strong), a:ASCII, 
>              l:Latin, r:Roman
>              to input: type "C-x 8 RET 6e" or "C-x 8 RET LATIN 
>              SMALL LETTER N"
>           buffer code: #x6E
>             file code: #x6E (encoded by coding system utf-8-unix)
>               display: composed to form "n̩" (see below)
> 
> Composed with the following character(s) "̩" using this font:
>   xft:-PfEd-Linux Libertine 
>   O-normal-normal-normal-*-21-*-*-*-*-0-iso10646-1
> by these glyphs:
>   [0 1 0 2420 13 0 13 10 1 nil]
>   [0 1 809 745 0 -9 -6 -1 5 [-2 -1 0]]
> 
> Character code properties: customize what to show
>   name: LATIN SMALL LETTER N
>   general-category: Ll (Letter, Lowercase)
>   decomposition: (110) ('n')
> 
> There are text properties here:
>   fontified            t
>   wrap-prefix          ""
> 
> ==============================
> 
> And with DejaVu Sans Mono (displaying the character correctly):
> 
> ==============================
> 
> 
>              position: 182 of 268 (68%), column: 5
>             character: n (displayed as n) (codepoint 110, #o156, 
>             #x6e)
>     preferred charset: ascii (ASCII (ISO646 IRV))
> code point in charset: 0x6E
>                script: latin
>                syntax: w 	which means: word
>              category: .:Base, L:Left-to-right (strong), a:ASCII, 
>              l:Latin, r:Roman
>              to input: type "C-x 8 RET 6e" or "C-x 8 RET LATIN 
>              SMALL LETTER N"
>           buffer code: #x6E
>             file code: #x6E (encoded by coding system utf-8-unix)
>               display: composed to form "n̩" (see below)
> 
> Composed with the following character(s) "̩" using this font:
>   xft:-PfEd-DejaVu Sans 
>   Mono-normal-normal-normal-*-15-*-*-*-m-0-iso10646-1
> by these glyphs:
>   [0 1 110 81 9 1 8 8 5 nil]
>   [0 1 809 689 0 4 6 -1 5 [-10 -1 0]]
> 
> Character code properties: customize what to show
>   name: LATIN SMALL LETTER N
>   general-category: Ll (Letter, Lowercase)
>   decomposition: (110) ('n')
> 
> There are text properties here:
>   fontified            t
>   wrap-prefix          ""
> 
> ==============================
> 
> The only difference I notice is with the numbers following "by 
> these glyphs", but I have no idea what those numbers mean.

(If you _really_ want to know what they mean, see the description of
GLYPH in the doc string of composition-get-gstring.)

Note how the first glyph of the composition data with the "bad" font
has zero as its 3rd component, whereas the "good" font has 110 there,
which is the codepoint of 'n'.  Having zero there is not really
normal, I think.

I downloaded the Libertine O font and tried that on MS-Windows.  I
don't see this problem there with that font.  So based on that, and on
the fact that another font displays this combination correctly on your
system, my primary suspect is neither Emacs nor the font, but the
shaping engine we use to compose characters.  On GNU/Linux we use
libm17n-flt as the shaping engine, so the first step is to make sure
you have its latest version installed.  I think the latest version is
only available in the CVS repository, which you can access through
Savannah:

   http://savannah.nongnu.org/projects/m17n/

If installing the latest version doesn't help, it might be a bug in
libm17n-flt, so I'm CC'ing Handa-san who might be able to help fix
this.

Re: Face issue -- possible bug

[Joost Kremers]

On Tue, Jul 17 2018, Eli Zaretskii wrote:
> Note how the first glyph of the composition data with the "bad" 
> font
> has zero as its 3rd component, whereas the "good" font has 110 
> there,
> which is the codepoint of 'n'.  Having zero there is not really
> normal, I think.
>
> I downloaded the Libertine O font and tried that on MS-Windows. 
> I
> don't see this problem there with that font.  So based on that, 
> and on
> the fact that another font displays this combination correctly 
> on your
> system, my primary suspect is neither Emacs nor the font, but 
> the
> shaping engine we use to compose characters.  On GNU/Linux we 
> use
> libm17n-flt as the shaping engine, so the first step is to make 
> sure
> you have its latest version installed.  I think the latest 
> version is
> only available in the CVS repository, which you can access 
> through
> Savannah:
>
>    http://savannah.nongnu.org/projects/m17n/
>
> If installing the latest version doesn't help, it might be a bug 
> in
> libm17n-flt, so I'm CC'ing Handa-san who might be able to help 
> fix
> this.

I'm actually having trouble accessing the sources. The invocation

cvs -z3 -d:pserver:anonymous@cvs.savannah.nongnu.org:/sources/m17n 
co m17n

as described at <http://savannah.nongnu.org/cvs/?group=m17n>

just gives me an empty CVS repository. Substituting `m17n-lib' for 
`m17n' does seem to checkout the sources, but the newest source 
files there are from 2014. (Which is why I said "seem to check out 
the sources", but I have no experience with CVS, so I may be doing 
something silly.)

According to <https://www.nongnu.org/m17n/>, that would correspond 
to version 1.17, released in 2014, which also happens to be the 
version installed on my Linux system (which is based on Ubuntu 
16.04). That www page also lists a version 1.18, from February of 
this year. I can try and install that, if you think it makes 
sense.

(Actually, the package installed on my system is called libm17n, 
not libm17n-flt, not sure if that matters.)

TIA

-- 
Joost Kremers
Life has its moments

Re: Face issue -- possible bug

[Eli Zaretskii]

> From: Joost Kremers <joostkremers@fastmail.fm>
> Date: Tue, 17 Jul 2018 23:23:08 +0200
> Cc: Kenichi Handa <handa@gnu.org>, emacs-devel@gnu.org
> 
> According to <https://www.nongnu.org/m17n/>, that would correspond 
> to version 1.17, released in 2014, which also happens to be the 
> version installed on my Linux system (which is based on Ubuntu 
> 16.04). That www page also lists a version 1.18, from February of 
> this year. I can try and install that, if you think it makes 
> sense.

(You mean 1.7.0 and 1.8.0, I guess, there's no 1.17 and 1.18.)  Yes, I
think it would be a good idea to install the latest versions.  Also of
libotf, version 0.9.16.  Let's see if this fixes the problem.

Thanks.

Re: Face issue -- possible bug

[Joost Kremers]

Hi Eli,

Sorry for the late reply on this issue, I've only now found the 
time to compile the newer m17n and libotf and then recompile 
Emacs.

On Wed, Jul 18 2018, Eli Zaretskii wrote:
> (You mean 1.7.0 and 1.8.0, I guess, there's no 1.17 and 1.18.)

Yes, indeed.

>  Yes, I
> think it would be a good idea to install the latest versions. 
> Also of
> libotf, version 0.9.16.  Let's see if this fixes the problem.

It doesn't seem to, unfortunately. I still get the small caps n as 
soon as the combining diacritic is added.

I removed the package m17n-0, m17n-dev, libotf0, libotf-dev and 
libotf-bin from my (Ubuntu 16.04-based) system, downloaded the 
source packages for m17n-lib, m17n-db (not sure if that's needed) 
and libotf, compiled and installed them (to /usr/local/), then 
recompiled Emacs.

Is there anything else I can try?

Thanks,

Joost


-- 
Joost Kremers
Life has its moments

Re: Face issue -- possible bug

[Eli Zaretskii]

> From: Joost Kremers <joostkremers@fastmail.fm>
> Cc: handa@gnu.org, emacs-devel@gnu.org
> Date: Mon, 13 Aug 2018 23:53:51 +0200
> 
> >  Yes, I
> > think it would be a good idea to install the latest versions. 
> > Also of
> > libotf, version 0.9.16.  Let's see if this fixes the problem.
> 
> It doesn't seem to, unfortunately. I still get the small caps n as 
> soon as the combining diacritic is added.
> 
> I removed the package m17n-0, m17n-dev, libotf0, libotf-dev and 
> libotf-bin from my (Ubuntu 16.04-based) system, downloaded the 
> source packages for m17n-lib, m17n-db (not sure if that's needed) 
> and libotf, compiled and installed them (to /usr/local/), then 
> recompiled Emacs.
> 
> Is there anything else I can try?

No, I think you did all you could.  I hope Handa-san will respond and
help us investigate further.

Re: Face issue -- possible bug

[Eli Zaretskii]

Ping!

> Date: Tue, 14 Aug 2018 05:29:08 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: handa@gnu.org, emacs-devel@gnu.org
> 
> > From: Joost Kremers <joostkremers@fastmail.fm>
> > Cc: handa@gnu.org, emacs-devel@gnu.org
> > Date: Mon, 13 Aug 2018 23:53:51 +0200
> > 
> > >  Yes, I
> > > think it would be a good idea to install the latest versions. 
> > > Also of
> > > libotf, version 0.9.16.  Let's see if this fixes the problem.
> > 
> > It doesn't seem to, unfortunately. I still get the small caps n as 
> > soon as the combining diacritic is added.
> > 
> > I removed the package m17n-0, m17n-dev, libotf0, libotf-dev and 
> > libotf-bin from my (Ubuntu 16.04-based) system, downloaded the 
> > source packages for m17n-lib, m17n-db (not sure if that's needed) 
> > and libotf, compiled and installed them (to /usr/local/), then 
> > recompiled Emacs.
> > 
> > Is there anything else I can try?
> 
> No, I think you did all you could.  I hope Handa-san will respond and
> help us investigate further.