bug#20984: 25.0.50; Combining accents don't display properly in certain fonts

bug#20984: 25.0.50; Combining accents don't display properly in certain fonts

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 690 bytes --]

Start emacs -Q then evaluate the following in the *scratch* (assuming
you have this font installed)

(set-face-attribute 'default nil :font "SourceCodePro Medium")
(insert (format "a%c" 768))

The result should look like "à", but instead, it looks like "a`" where
the letter "a" is actually distorted and shifted up (image attached).
The same happens with other combining symbols.

On the other hand, if I type a second letter "a", then the grave wil be
placed over it, while the first letter stays distorted.

Again, this is easier to understand with the attached image. The
second and third lines of the image show the weird behavior. The first
line is correct, I think.

[-- Attachment #2: screenshot-2015-07-05--162913.png --]
[-- Type: image/png, Size: 4300 bytes --]

bug#20984: 25.0.50; Combining accents don't display properly in certain fonts

[Eli Zaretskii]

> Date: Sun, 5 Jul 2015 17:24:32 +0100
> From: Artur Malabarba <bruce.connor.am@gmail.com>
> 
> Start emacs -Q then evaluate the following in the *scratch* (assuming
> you have this font installed)
> 
> (set-face-attribute 'default nil :font "SourceCodePro Medium")
> (insert (format "a%c" 768))
> 
> The result should look like "à", but instead, it looks like "a`" where
> the letter "a" is actually distorted and shifted up (image attached).
> The same happens with other combining symbols.

Do both glyphs come from the same font, according to "C-u C-x ="?  (I
don't have that font installed to try that myself.)

Emacs can only compose characters if their glyphs are covered by the
same font.

bug#20984: 25.0.50; Combining accents don't display properly in certain fonts

[Artur Malabarba]

2015-07-05 17:33 GMT+01:00 Eli Zaretskii <eliz@gnu.org>:
>> Date: Sun, 5 Jul 2015 17:24:32 +0100
>> From: Artur Malabarba <bruce.connor.am@gmail.com>
>>
>> Start emacs -Q then evaluate the following in the *scratch* (assuming
>> you have this font installed)
>>
>> (set-face-attribute 'default nil :font "SourceCodePro Medium")
>> (insert (format "a%c" 768))
>>
>> The result should look like "à", but instead, it looks like "a`" where
>> the letter "a" is actually distorted and shifted up (image attached).
>> The same happens with other combining symbols.
>
> Do both glyphs come from the same font, according to "C-u C-x ="?  (I
> don't have that font installed to try that myself.)
>
> Emacs can only compose characters if their glyphs are covered by the
> same font.

Yes. Investigating the weird combination yields this on the description buffer:

Composed with the following character(s) "̀" using this font:
  xft:-adobe-Source Code Pro-normal-normal-normal-*-13-*-*-*-m-0-iso10646-1
by these glyphs:
  [0 1 0 626 8 1 6 8 -3 nil]
  [0 1 0 734 0 1 5 11 -9 [0 0 0]]

And investigating the characters individually yields the same font for both

    xft:-adobe-Source Code
Pro-normal-normal-normal-*-13-*-*-*-m-0-iso10646-1 (#x1C)
    xft:-adobe-Source Code
Pro-normal-normal-normal-*-13-*-*-*-m-0-iso10646-1 (#x2DD)

Should I paste the full *help* buffer here?

bug#20984: 25.0.50; Combining accents don't display properly in certain fonts

[Eli Zaretskii]

> Date: Sun, 5 Jul 2015 17:42:45 +0100
> From: Artur Malabarba <bruce.connor.am@gmail.com>
> Cc: 20984@debbugs.gnu.org
> 
> > Do both glyphs come from the same font, according to "C-u C-x ="?  (I
> > don't have that font installed to try that myself.)
> >
> > Emacs can only compose characters if their glyphs are covered by the
> > same font.
> 
> Yes. Investigating the weird combination yields this on the description buffer:
> 
> Composed with the following character(s) "̀" using this font:
>   xft:-adobe-Source Code Pro-normal-normal-normal-*-13-*-*-*-m-0-iso10646-1
> by these glyphs:
>   [0 1 0 626 8 1 6 8 -3 nil]
>   [0 1 0 734 0 1 5 11 -9 [0 0 0]]
> 
> And investigating the characters individually yields the same font for both
> 
>     xft:-adobe-Source Code
> Pro-normal-normal-normal-*-13-*-*-*-m-0-iso10646-1 (#x1C)
>     xft:-adobe-Source Code
> Pro-normal-normal-normal-*-13-*-*-*-m-0-iso10646-1 (#x2DD)
> 
> Should I paste the full *help* buffer here?

There's no need, thanks.  But it would be interesting, I think, to see
the output of "C-u C-x =" for the same pair of inserted characters,
but with your default font (assuming it is capable of displaying the
accented a).

Anyway, I've tried this on my system, after installing version 1.020
of the font, and I don't see the problem you report: the 2 characters
are composed and displayed as you'd expect.

So I think this might be some issue with the shaping engine you are
using -- do you have the latest versions of the libraries mentioned in
INSTALL, under "Complex Text Layout support libraries"?  If you do,
perhaps Handa-san could comment on this.

For the record, my system uses the Windows standard Uniscribe shaping
engine, and the composition information it returns is different:

  Composed with the following character(s) "̀" using this font:
    uniscribe:-outline-Source Code Pro Medium-normal-normal-normal-mono-13-*-*-*-c-*-iso8859-1
  by these glyphs:
    [0 1 97 231 8 1 7 12 4 nil]

bug#20984: 25.0.50; Combining accents don't display properly in certain fonts

[Artur Malabarba]

>> Should I paste the full *help* buffer here?
>
> There's no need, thanks.  But it would be interesting, I think, to see
> the output of "C-u C-x =" for the same pair of inserted characters,
> but with your default font (assuming it is capable of displaying the
> accented a).
>

With my default font (DejaVu Sans Mono), which does correctly display
the grave a, describing the combined characters gives the following

Composed with the following character(s) "̀" using this font:
  xft:-unknown-DejaVu Sans Mono-normal-normal-normal-*-15-*-*-*-m-0-iso10646-1
by these glyphs:
  [0 1 97 68 9 1 8 11 0 nil]
  [0 1 768 648 0 2 6 12 -9 [-9 0 0]]

And describing them individually gives the following

xft:-unknown-DejaVu Sans
Mono-normal-normal-normal-*-15-*-*-*-m-0-iso10646-1 (#x44)
xft:-unknown-DejaVu Sans
Mono-normal-normal-normal-*-15-*-*-*-m-0-iso10646-1 (#x288)

> Anyway, I've tried this on my system, after installing version 1.020
> of the font, and I don't see the problem you report: the 2 characters
> are composed and displayed as you'd expect.
>
> So I think this might be some issue with the shaping engine you are
> using -- do you have the latest versions of the libraries mentioned in
> INSTALL, under "Complex Text Layout support libraries"?  If you do,
> perhaps Handa-san could comment on this.

Out of the 3 mentioned there ("m17n-db", "libm17n-flt", "libotf")
- I see the following 2 installed (and up-to-date) on my system
(arch): m17n-db, m17n-lib, libotf.
- Looking through my emacs's config.log, I see reports that the
following two were detected: M17N_FLT, LIBOTF.

bug#20984: 25.0.50; Combining accents don't display properly in certain fonts

[handa]

In article <83fv52wnb8.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> > Yes. Investigating the weird combination yields this on the description buffer:
> > Composed with the following character(s) "̀" using this font:
> >   xft:-adobe-Source Code Pro-normal-normal-normal-*-13-*-*-*-m-0-iso10646-1
> > by these glyphs:
> >   [0 1 0 626 8 1 6 8 -3 nil]
> >   [0 1 0 734 0 1 5 11 -9 [0 0 0]]

I tried that font on my Lubuntu system with Emacs 24.5 and the latest
trunk version.  I see no problem with 24.5, i.e. a-grave is correctly
displayed, but see the same problem as yours with trunk version.

So, it seems that something was broken recently.

---
K. Handa
handa@gnu.org

bug#20984: 25.0.50; Combining accents don't display properly in certain fonts

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 459 bytes --]

> I tried that font on my Lubuntu system with Emacs 24.5 and the latest
> trunk version.  I see no problem with 24.5, i.e. a-grave is correctly
> displayed, but see the same problem as yours with trunk version.
>
> So, it seems that something was broken recently.

That's strange, I tested on a 24.2 build I had lying around and found the
same problem. Though I can't even remember how/when that was built.

I'll check out 24.5 tomorrow and do a proper test.

[-- Attachment #2: Type: text/html, Size: 561 bytes --]

bug#20984: 25.0.50; Combining accents don't display properly in certain fonts

[Eli Zaretskii]

> From: handa <handa@gnu.org>
> Cc: bruce.connor.am@gmail.com, 20984@debbugs.gnu.org
> Date: Mon, 06 Jul 2015 23:59:14 +0900
> 
> > >   [0 1 0 626 8 1 6 8 -3 nil]
> > >   [0 1 0 734 0 1 5 11 -9 [0 0 0]]
> 
> I tried that font on my Lubuntu system with Emacs 24.5 and the latest
> trunk version.  I see no problem with 24.5, i.e. a-grave is correctly
> displayed, but see the same problem as yours with trunk version.
> 
> So, it seems that something was broken recently.

Could it be my latest changes related to fontsets?  Though then the
Windows build should have been affected as well.

The composition data above looks wrong, doesn't it?  Like if the 'a'
part of the grapheme cluster comes from some different codepoint, not
from 'a', am I right?

bug#20984: 25.0.50; Combining accents don't display properly in certain fonts

[handa]

In article <831tglw6j8.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> > From: handa <handa@gnu.org>
> > Cc: bruce.connor.am@gmail.com, 20984@debbugs.gnu.org
> > Date: Mon, 06 Jul 2015 23:59:14 +0900
> > 
> > > >   [0 1 0 626 8 1 6 8 -3 nil]
> > > >   [0 1 0 734 0 1 5 11 -9 [0 0 0]]
> > 
> > I tried that font on my Lubuntu system with Emacs 24.5 and the latest
> > trunk version.  I see no problem with 24.5, i.e. a-grave is correctly
> > displayed, but see the same problem as yours with trunk version.
> > 
> > So, it seems that something was broken recently.

> Could it be my latest changes related to fontsets?  Though then the
> Windows build should have been affected as well.

I don't think so.  Your change is related to font selection, and in the
current case, the font is correctly selected.

> The composition data above looks wrong, doesn't it?  Like if the 'a'
> part of the grapheme cluster comes from some different codepoint, not
> from 'a', am I right?

The above data is completely broken.  The 3rd element of each vector is
a character code, and it should be 97 and 768 respectively.

In my case too, the latest trunk Emacs returns the same vector as above,
and, Emacs 24.5 returns this:

  [0 1 97 28 8 -1 8 7 0 nil]
  [0 1 768 733 8 1 6 10 -8 [-8 0 0]]

---
K. Handa
handa@gnu.org

bug#20984: 25.0.50; Combining accents don't display properly in certain fonts

[Eli Zaretskii]

> From: handa <handa@gnu.org>
> Cc: bruce.connor.am@gmail.com, 20984@debbugs.gnu.org
> Date: Thu, 09 Jul 2015 18:27:08 +0900
> 
> > > > >   [0 1 0 626 8 1 6 8 -3 nil]
> > > > >   [0 1 0 734 0 1 5 11 -9 [0 0 0]]
> > > 
> > The composition data above looks wrong, doesn't it?  Like if the 'a'
> > part of the grapheme cluster comes from some different codepoint, not
> > from 'a', am I right?
> 
> The above data is completely broken.  The 3rd element of each vector is
> a character code, and it should be 97 and 768 respectively.
> 
> In my case too, the latest trunk Emacs returns the same vector as above,
> and, Emacs 24.5 returns this:
> 
>   [0 1 97 28 8 -1 8 7 0 nil]
>   [0 1 768 733 8 1 6 10 -8 [-8 0 0]]

Can you point me to the code that is involved in creation of the
composition data, when the font back-end is xftfont?  Specifically,
those portions of creating the composition data that depend on the
font features, since the same composition displays correctly for the
OP (and I guess for you as well) with DejaVu Sans Mono.  I will then
try to see what changes were done there since 24.5.

Thanks.

bug#20984: 25.0.50; Combining accents don't display properly in certain fonts

[Artur Malabarba]

> OP (and I guess for you as well) with DejaVu Sans Mono.  I will then
> try to see what changes were done there since 24.5.

Just to add information. I built the emacs-24.5 tag today and I still
see the same issue on it. So don't be surprised if the answer isn't
there. =/

bug#20984: 25.0.50; Combining accents don't display properly in certain fonts

[Eli Zaretskii]

> Date: Thu, 9 Jul 2015 22:55:09 +0100
> From: Artur Malabarba <bruce.connor.am@gmail.com>
> Cc: Kenichi Handa <handa@gnu.org>, 20984@debbugs.gnu.org
> 
> > OP (and I guess for you as well) with DejaVu Sans Mono.  I will then
> > try to see what changes were done there since 24.5.
> 
> Just to add information. I built the emacs-24.5 tag today and I still
> see the same issue on it. So don't be surprised if the answer isn't
> there. =/

Yes, I understand.  But walking through the code might be a good
starting point anyway, since the composition data is so completely
broken.

Btw, I wonder why xftfont and libotf don't do the same as Uniscribe,
i.e. instruct Emacs to display the à character.  This is what the
composition data I see here says:

  [0 1 97 231 8 1 7 12 4 nil]

This single vector tells Emacs to display one glyph, and 231 is its
code in the font.

It is strange that libotf doesn't take this shortcut, although I'm
quite sure a glyph for à is available both in DejaVu Sans Mono and in
Source Code Pro.  But I don't know enough about libotf.

bug#20984: 25.0.50; Combining accents don't display properly in certain fonts

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 241 bytes --]

> It is strange that libotf doesn't take this shortcut, although I'm
> quite sure a glyph for à is available both in DejaVu Sans Mono and in
> Source Code Pro.

Indeed it is. If I type out the actual letter (grave a) it displays fine.

[-- Attachment #2: Type: text/html, Size: 301 bytes --]

bug#20984: 25.0.50; Combining accents don't display properly in certain fonts

[handa]

[-- Attachment #1: Type: text/plain, Size: 1618 bytes --]

In article <83oajkbkbo.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> Btw, I wonder why xftfont and libotf don't do the same as Uniscribe,
> i.e. instruct Emacs to display the à character.  This is what the
> composition data I see here says:

>   [0 1 97 231 8 1 7 12 4 nil]

> This single vector tells Emacs to display one glyph, and 231 is its
> code in the font.

> It is strange that libotf doesn't take this shortcut, although I'm
> quite sure a glyph for à is available both in DejaVu Sans Mono and in
> Source Code Pro.  But I don't know enough about libotf.

Sorry, I found that my build of emacs-24.5 was without m17n-flt (and
libotf).  So, the combining was done by Emacs itself using the function
compose-gstring-for-graphic.

I've just rebuild emacs-24.5 with m17n-flt, and see the same problem as
the trunk, which means that the culprit may be in m17n-flt or libotf.

So, I checked m17n-flt and found that the rule for combining latin
characters has a bug when a font has such OTF features as subs, sups.

Please try the attached COMBINING.flt by these steps:
1. make the directory ~/.m17n.d
2. put COMBINING.flt under that directory.
3. run emacs

By the why, the reason of m17n-flt/libotf not taking the shortcut above
is that the Source Code Pro font doesn't have such a feature.  I suspect
Uniscribe has a special code for using precomposed glyph without asking
a font about its features.  So, perhaps, even with TTF font (i.e. a font
of no OTF features), Uniscribe can display a-grave sequence with the
precomposed glyph.

---
K. Handa
handa@gnu.org


[-- Attachment #2: COMBINING.flt --]
[-- Type: application/octet-stream, Size: 2075 bytes --]

;; COMBINING.flt -- Font Layout Table for combining diacritical marks
;; Copyright (C) 2007  AIST (H15PRO112)
;; See the end for copying conditions.

(font layouter combining nil)

;;; <li> COMBINING.flt
;;;
;;; For combining diacritical marsk (U+0300..U+036F).

(category
 ;; The contents is build up by the m17n-lib.
 )

(generator
 (0
  (cond
   ("(u)([a-t]+)"
    (cond
     ((font-facility :otf=DFLT+mark) < :otf=DFLT=+mark,mkmk >)
     (".*"
      <	=				; combining class
      (cond ("a" Bc.Bc =)		; < 200
	    ("b" bl.tc =)		; 200
	    ("c" bc.tc =)		; 202
	    ("d" br.tc =)		; 204
	    ("e" Bl.Br =)		; 208
	    ("f" Br.Bl =)		; 210
	    ("g" tl.bc =)		; 212
	    ("h" tc.bc =)		; 214
	    ("i" tr.bc =)		; 216
	    ("j" bl.tc =)		; 218
	    ("k" bc-tc =)		; 220
	    ("l" br-tc =)		; 222
	    ("m" Bl.Br =)		; 224
	    ("n" Br.Bl =)		; 226
	    ("o" tl+bc =)		; 228
	    ("p" tc+bc =)		; 230
	    ("q" tr+bc =)		; 232
	    ("r" br-tr =)		; 233
	    ("s" tr+br =)		; 234
	    ("t" bc-tc =))		; 240
      * >)))
   ("[a-t]" [ = ])
   ("." =))
  *))

;; Copyright (C) 2007
;;   National Institute of Advanced Industrial Science and Technology (AIST)
;;   Registration Number H15PRO112

;; This file is part of the m17n database; a sub-part of the m17n
;; library.

;; The m17n library is free software; you can redistribute it and/or
;; modify it under the terms of the GNU Lesser General Public License
;; as published by the Free Software Foundation; either version 2.1 of
;; the License, or (at your option) any later version.

;; The m17n library is distributed in the hope that it will be useful,
;; but WITHOUT ANY WARRANTY; without even the implied warranty of
;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
;; Lesser General Public License for more details.

;; You should have received a copy of the GNU Lesser General Public
;; License along with the m17n library; if not, write to the Free
;; Software Foundation, Inc., 51 Franklin Street, Fifth Floor,
;; Boston, MA 02110-1301, USA.

;; Local Variables:
;; mode: emacs-lisp
;; End:

bug#20984: 25.0.50; Combining accents don't display properly in certain fonts

[Artur Malabarba]

Indeed, this works.

2015-07-10 17:06 GMT+01:00 handa <handa@gnu.org>:
> In article <83oajkbkbo.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:
>
>> Btw, I wonder why xftfont and libotf don't do the same as Uniscribe,
>> i.e. instruct Emacs to display the à character.  This is what the
>> composition data I see here says:
>
>>   [0 1 97 231 8 1 7 12 4 nil]
>
>> This single vector tells Emacs to display one glyph, and 231 is its
>> code in the font.
>
>> It is strange that libotf doesn't take this shortcut, although I'm
>> quite sure a glyph for à is available both in DejaVu Sans Mono and in
>> Source Code Pro.  But I don't know enough about libotf.
>
> Sorry, I found that my build of emacs-24.5 was without m17n-flt (and
> libotf).  So, the combining was done by Emacs itself using the function
> compose-gstring-for-graphic.
>
> I've just rebuild emacs-24.5 with m17n-flt, and see the same problem as
> the trunk, which means that the culprit may be in m17n-flt or libotf.
>
> So, I checked m17n-flt and found that the rule for combining latin
> characters has a bug when a font has such OTF features as subs, sups.
>
> Please try the attached COMBINING.flt by these steps:
> 1. make the directory ~/.m17n.d
> 2. put COMBINING.flt under that directory.
> 3. run emacs
>
> By the why, the reason of m17n-flt/libotf not taking the shortcut above
> is that the Source Code Pro font doesn't have such a feature.  I suspect
> Uniscribe has a special code for using precomposed glyph without asking
> a font about its features.  So, perhaps, even with TTF font (i.e. a font
> of no OTF features), Uniscribe can display a-grave sequence with the
> precomposed glyph.
>
> ---
> K. Handa
> handa@gnu.org
>

bug#20984: 25.0.50; Combining accents don't display properly in certain fonts

[Eli Zaretskii]

> Date: Wed, 15 Jul 2015 20:05:20 +0100
> From: Artur Malabarba <bruce.connor.am@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>, 20984@debbugs.gnu.org
> 
> Indeed, this works.

Thanks for testing.

So can we close this bug, or are there any leftovers?

bug#20984: 25.0.50; Combining accents don't display properly in certain fonts

[Artur Malabarba]

[-- Attachment #1: Type: text/plain, Size: 82 bytes --]

> So can we close this bug, or are there any leftovers?

Yes. It's enough for me.

[-- Attachment #2: Type: text/html, Size: 123 bytes --]

bug#20984: 25.0.50; Combining accents don't display properly in certain fonts

[Eli Zaretskii]

> Date: Wed, 15 Jul 2015 23:29:20 +0100
> From: Artur Malabarba <bruce.connor.am@gmail.com>
> Cc: 20984@debbugs.gnu.org, Kenichi Handa <handa@gnu.org>
> 
> > So can we close this bug, or are there any leftovers?
> 
> Yes. It's enough for me. 

Thanks, closing.

bug#20984: 25.0.50; Combining accents don't display properly in certain fonts

[handa]

In article <CAAdUY-JQoRXbdHT3rERAVhfCb76cVViBD3i5y85dYR+sL5mqbA@mail.gmail.com>, Artur Malabarba <bruce.connor.am@gmail.com> writes:

> Indeed, this works.

Thank you for testing.  The next release of m17n-db will come with that
new version.

---
K. Handa
handa@gnu.org


> 2015-07-10 17:06 GMT+01:00 handa <handa@gnu.org>:
> > In article <83oajkbkbo.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:
> >
>>> Btw, I wonder why xftfont and libotf don't do the same as Uniscribe,
>>> i.e. instruct Emacs to display the à character.  This is what the
>>> composition data I see here says:
> >
>>> [0 1 97 231 8 1 7 12 4 nil]
> >
>>> This single vector tells Emacs to display one glyph, and 231 is its
>>> code in the font.
> >
>>> It is strange that libotf doesn't take this shortcut, although I'm
>>> quite sure a glyph for à is available both in DejaVu Sans Mono and in
>>> Source Code Pro.  But I don't know enough about libotf.
> >
> > Sorry, I found that my build of emacs-24.5 was without m17n-flt (and
> > libotf).  So, the combining was done by Emacs itself using the function
> > compose-gstring-for-graphic.
> >
> > I've just rebuild emacs-24.5 with m17n-flt, and see the same problem as
> > the trunk, which means that the culprit may be in m17n-flt or libotf.
> >
> > So, I checked m17n-flt and found that the rule for combining latin
> > characters has a bug when a font has such OTF features as subs, sups.
> >
> > Please try the attached COMBINING.flt by these steps:
> > 1. make the directory ~/.m17n.d
> > 2. put COMBINING.flt under that directory.
> > 3. run emacs
> >
> > By the why, the reason of m17n-flt/libotf not taking the shortcut above
> > is that the Source Code Pro font doesn't have such a feature.  I suspect
> > Uniscribe has a special code for using precomposed glyph without asking
> > a font about its features.  So, perhaps, even with TTF font (i.e. a font
> > of no OTF features), Uniscribe can display a-grave sequence with the
> > precomposed glyph.
> >
> > ---
> > K. Handa
> > handa@gnu.org
> >