bug#23292: 24.5; Combining characters do not reliably combine

bug#23292: 24.5; Combining characters do not reliably combine

[Honore Doktorr]

Combining characters often do not render combined in emacs 24.5.
For example, the combining short solidus overlay (0x337, o̷) appears
following the character it's meant to combine with---although they are
both highlighted together and so forth.

cf. https://i.imgsafe.org/8369941.jpeg for a visual reference.

This seems similar to bug 17261, but both characters are from the same
font, DejaVu Sans Mono.

describe-char for the combined (but separately rendered) o̷:
---------------------------------------------------------------------------
             position: 14 of 70 (19%), column: 12
            character: o (displayed as o) (codepoint 111, #o157, #x6f)
    preferred charset: ascii (ASCII (ISO646 IRV))
code point in charset: 0x6F
               script: latin
               syntax: w     which means: word
             category: .:Base, L:Left-to-right (strong), a:ASCII,
l:Latin, r:Roman
             to input: type "C-x 8 RET HEX-CODEPOINT" or "C-x 8 RET NAME"
          buffer code: #x6F
            file code: #x6F (encoded by coding system utf-8)
              display: composed to form "o̷" (see below)

Composed with the following character(s) "̷" using this font:
  xft:-PfEd-DejaVu Sans Mono-normal-normal-normal-*-13-*-*-*-m-0-iso10646-1
by these glyphs:
  [0 1 111 82 8 1 7 7 0 nil]
  [0 1 823 703 8 0 8 8 1 nil]

Character code properties: customize what to show
  name: LATIN SMALL LETTER O
  general-category: Ll (Letter, Lowercase)
  decomposition: (111) ('o')
---------------------------------------------------------------------------

describe-char for the combining short solidus overlay by itself:
---------------------------------------------------------------------------
            position: 619 of 1008 (61%), column: 42
            character: ̷ (displayed as ̷) (codepoint 823, #o1467, #x337)
    preferred charset: unicode-bmp (Unicode Basic Multilingual Plane
(U+0000..U+FFFF))
code point in charset: 0x0337
               script: latin
               syntax: w     which means: word
             category: ^:Combining
             to input: type "C-x 8 RET HEX-CODEPOINT" or "C-x 8 RET NAME"
          buffer code: #xCC #xB7
            file code: #xCC #xB7 (encoded by coding system utf-8)
              display: composed to form "̷" (see below)

Composed by the rule:
    (TAB ?̷ TAB)
The component character(s) are displayed by these fonts (glyph codes):
 ̷: xft:-PfEd-DejaVu Sans
Mono-normal-normal-normal-*-13-*-*-*-m-0-iso10646-1 (#x2BF)
See the variable `reference-point-alist' for the meaning of the rule.

Character code properties: customize what to show
  name: COMBINING SHORT SOLIDUS OVERLAY
  old-name: NON-SPACING SHORT SLASH OVERLAY
  general-category: Mn (Mark, Nonspacing)
  decomposition: (823) ('̷')

There are text properties here:
  composition          [Show]
---------------------------------------------------------------------------




In GNU Emacs 24.5.1 (x86_64-redhat-linux-gnu, GTK+ Version 3.18.7)
 of 2016-02-03 on buildhw-05.phx2.fedoraproject.org
Windowing system distributor `Fedora Project', version 11.0.11803000
System Description:    Fedora release 23 (Twenty Three)

Configured using:
 `configure --build=x86_64-redhat-linux-gnu
 --host=x86_64-redhat-linux-gnu --program-prefix=
 --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr
 --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc
 --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64
 --libexecdir=/usr/libexec --localstatedir=/var
 --sharedstatedir=/var/lib --mandir=/usr/share/man
 --infodir=/usr/share/info --with-dbus --with-gif --with-jpeg --with-png
 --with-rsvg --with-tiff --with-xft --with-xpm --with-x-toolkit=gtk3
 --with-gpm=no build_alias=x86_64-redhat-linux-gnu
 host_alias=x86_64-redhat-linux-gnu 'CFLAGS=-DMAIL_USE_LOCKF -O2 -g
 -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2
 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4
 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1
 -m64 -mtune=generic' LDFLAGS=-Wl,-z,relro'

Important settings:
  value of $LANG: en_US.utf8
  value of $XMODIFIERS: @im=none
  locale-coding-system: utf-8-unix

Major mode: Lisp Interaction

Minor modes in effect:
  show-paren-mode: t
  delete-selection-mode: t
  display-time-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  temp-buffer-resize-mode: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent messages:
Mark set [2 times]
Type "q" to restore previous buffer, M-x scroll-up to scroll help.
byte-code: End of buffer
<mouse-6> is undefined
byte-code: Beginning of buffer [4 times]
Quit
<<< Type SPC or RET to bury the buffer list >>>

byte-code: Beginning of buffer [2 times]


Load-path shadows:
/home/denis/.emacs.d/elpa/js2-mode-20141118/js2-mode hides
/home/denis/lib/site-lisp/…aside/js2-mode
~/lib/site-lisp/markdown-mode hides
/usr/share/emacs/site-lisp/goodies/markdown-mode
~/lib/site-lisp/css-mode hides /usr/share/emacs/24.5/lisp/textmodes/css-mode
/usr/share/emacs/site-lisp/gnus-bonus/nnir hides
/usr/share/emacs/24.5/lisp/gnus/nnir
/usr/share/emacs/site-lisp/gnus-bonus/nnnil hides
/usr/share/emacs/24.5/lisp/gnus/nnnil
/usr/share/emacs/site-lisp/gnus-bonus/spam-stat hides
/usr/share/emacs/24.5/lisp/gnus/spam-stat
~/lib/site-lisp/longlines hides /usr/share/emacs/24.5/lisp/obsolete/longlines

Features:
(shadow sort gnus-util mail-extr emacsbug message idna format-spec
rfc822 mml mml-sec mm-decode mm-bodies mm-encode mail-parse rfc2231
mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums
mm-util mail-prsvr mail-utils pp mule-util ebuff-menu wid-edit
descr-text help-mode package epg-config server follow-mouse desktop
frameset saveplace paren cl-macs cl gv apropos derived markdown-mode
cl-loaddefs cl-lib thingatpt noutline outline easymenu advice help-fns
mustache-mode delsel grep compile comint ansi-color ring cus-start
cus-load time 50magit emacs-goodies-loaddefs easy-mmode time-date
tooltip electric uniquify ediff-hook vc-hooks lisp-float-type mwheel
x-win x-dnd tool-bar dnd fontset image regexp-opt fringe tabulated-list
newcomment lisp-mode prog-mode register page menu-bar rfn-eshadow timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
frame cham georgian utf-8-lang misc-lang vietnamese tibetan thai
tai-viet lao korean japanese hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese case-table epa-hook jka-cmpr-hook help
simple abbrev minibuffer nadvice loaddefs button faces cus-face macroexp
files text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote make-network-process
dbusbind gfilenotify dynamic-setting system-font-setting
font-render-setting move-toolbar gtk x-toolkit x multi-tty emacs)

Memory information:
((conses 16 129620 19595)
 (symbols 48 32900 0)
 (miscs 40 1221 515)
 (strings 32 37972 9706)
 (string-bytes 1 1003542)
 (vectors 16 19979)
 (vector-slots 8 1305892 204196)
 (floats 8 78 475)
 (intervals 56 445 16)
 (buffers 960 16)
 (heap 1024 49802 1271))

bug#23292: 24.5; Combining characters do not reliably combine

[Eli Zaretskii]

> Date: Thu, 14 Apr 2016 15:26:03 -0400
> From: Honore Doktorr <hdfssk@gmail.com>
> 
> Combining characters often do not render combined in emacs 24.5.
> For example, the combining short solidus overlay (0x337, o̷) appears
> following the character it's meant to combine with---although they are
> both highlighted together and so forth.
> 
> cf. https://i.imgsafe.org/8369941.jpeg for a visual reference.
> 
> This seems similar to bug 17261, but both characters are from the same
> font, DejaVu Sans Mono.

Was your Emacs built with libotf and other libraries mentioned in
INSTALL under "Complex Text Layout"?

When I arrange for Emacs to use a font that supports both 'o' and the
solidus, the sequence you show does display as a single glyph.  (My
default font doesn't have the solidus, so I need that special
arrangement.)

The fact that Emacs displays a cursor on both of the characters
clearly shows that Emacs did combine them, but the font and/or the
shaping engine didn't display them overlaid.  Not sure why, perhaps
upgrade your libotf and related libraries?

bug#23292: 24.5; Combining characters do not reliably combine

[Eli Zaretskii]

[Please keep the bug address on the CC list.]

> Date: Thu, 14 Apr 2016 17:30:01 -0400
> From: Honore Doktorr <hdfssk@gmail.com>
> 
> > Was your Emacs built with libotf and other libraries mentioned in
> > INSTALL under "Complex Text Layout"?
> >
> > When I arrange for Emacs to use a font that supports both 'o' and the
> > solidus, the sequence you show does display as a single glyph. (My
> > default font doesn't have the solidus, so I need that special
> > arrangement.)
> >
> > The fact that Emacs displays a cursor on both of the characters
> > clearly shows that Emacs did combine them, but the font and/or the
> > shaping engine didn't display them overlaid. Not sure why, perhaps
> > upgrade your libotf and related libraries?
> 
> Yes, it looks like with the latest upstream versions of the complex text libraries/db:
> 
> $ ldd /usr/bin/emacs |egrep 'lib(otf|m17n-flt)'
> libotf.so.0 => /lib64/libotf.so.0 (0x00007f6cd2a05000)
> libm17n-flt.so.0 => /lib64/libm17n-flt.so.0 (0x00007f6cd25cc000)
> $ dnf -C list libotf m17n-lib m17n-db
> Last metadata expiration check: 0:12:20 ago on Thu Apr 14 17:09:12 2016.
> Installed Packages
> libotf.x86_64 0.9.13-6.fc23 @@commandline
> m17n-db.noarch 1.7.0-5.fc23 @@commandline
> m17n-lib.x86_64 1.7.0-4.fc23 @@commandline
> 
> I'm using the fedora 23 precompiled packages.

Does anyone else see this with DejaVu Sans Mono?

bug#23292: 24.5; Combining characters do not reliably combine

[Alexis]

Eli Zaretskii <eliz@gnu.org> writes:

> Does anyone else see this with DejaVu Sans Mono?

Not by doing `C-x 8 / o', which produces 'ø'; but is that the way 
i should be testing this?

bug#23292: 24.5; Combining characters do not reliably combine

[Eli Zaretskii]

> From: Alexis <flexibeast@gmail.com>
> Cc: Honore Doktorr <hdfssk@gmail.com>, 23292@debbugs.gnu.org
> Date: Fri, 15 Apr 2016 17:47:25 +1000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Does anyone else see this with DejaVu Sans Mono?
> 
> Not by doing `C-x 8 / o', which produces 'ø'; but is that the way 
> i should be testing this?

No.  Type 'o', then "C-x 8 RET 337".

bug#23292: 24.5; Combining characters do not reliably combine

[Alexis]

Eli Zaretskii <eliz@gnu.org> writes:

> No.  Type 'o', then "C-x 8 RET 337".

Done. i get the same result as the OP - i.e. the 'o' and the '/' 
are visually separated - using both DejaVu Sans Mono (Book) and 
Inconsolata-G (my default font).

bug#23292: 24.5; Combining characters do not reliably combine

[Eli Zaretskii]

> From: Alexis <flexibeast@gmail.com>
> Cc: hdfssk@gmail.com, 23292@debbugs.gnu.org
> Date: Fri, 15 Apr 2016 18:17:48 +1000
> 
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > No.  Type 'o', then "C-x 8 RET 337".
> 
> Done. i get the same result as the OP - i.e. the 'o' and the '/' 
> are visually separated - using both DejaVu Sans Mono (Book) and 
> Inconsolata-G (my default font).

Thanks.

This is strange.  I'm CC'ing Handa-san, perhaps there's some issue in
libm17n-flt or its database for this case.

For the record, the composition works for me on MS-Windows using the
Arial Unicode MS font, and the composition data looks quite different
(and makes much more sense to me) than what the OP shows:

  Composed with the following character(s) "̷" using this font:
    uniscribe:-outline-Arial Unicode MS-normal-normal-normal-sans-13-*-*-*-p-*-iso8859-1
  by these glyphs:
    [0 1 111 82 7 1 6 14 4 nil]
    [0 1 823 671 0 -5 -2 14 4 nil]

The OP said the composition data he gets is this:

  Composed with the following character(s) "̷" using this font:
    xft:-PfEd-DejaVu Sans Mono-normal-normal-normal-*-13-*-*-*-m-0-iso10646-1
  by these glyphs:
    [0 1 111 82 8 1 7 7 0 nil]
    [0 1 823 703 8 0 8 8 1 nil]

and the offsets in the second vector look wrong to me, FWIW.

bug#23292: 24.5; Combining characters do not reliably combine

[Alexis]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Alexis <flexibeast@gmail.com> Cc: hdfssk@gmail.com, 
>> 23292@debbugs.gnu.org Date: Fri, 15 Apr 2016 18:17:48 +1000
>> 
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > No.  Type 'o', then "C-x 8 RET 337".
>> 
>> Done. i get the same result as the OP - i.e. the 'o' and the 
>> '/'  are visually separated - using both DejaVu Sans Mono 
>> (Book) and  Inconsolata-G (my default font).

My apologies, i accidentally used a non -Q instance to test 
this. :-/ In a -Q instance, it turns out that DejaVu Sans Mono 
(Book) /does/ compose visually:

Composed with the following character(s) "̷" using this font: 
  xft:-unknown-DejaVu Sans 
  Mono-normal-normal-normal-*-13-*-*-*-m-0-iso10646-1 
by these glyphs: 
  [0 1 111 82 8 0 8 8 1 nil] [0 1 823 703 0 0 8 8 1 [-8 0 0]] 

Sorry!

However, Inconsolata-g definitely doesn't compose visually.

Interestingly, when i use `describe-char' on the 'o', i'm told 
that the glyph is sourced from Inconsolata-g, but when i use it on 
the '/', it tells me it's sourced from Gentium ....

bug#23292: 24.5; Combining characters do not reliably combine

[Eli Zaretskii]

> From: Alexis <flexibeast@gmail.com>
> Cc: Kenichi Handa <handa@gnu.org>, hdfssk@gmail.com, 23292@debbugs.gnu.org
> Date: Fri, 15 Apr 2016 19:03:15 +1000
> 
> My apologies, i accidentally used a non -Q instance to test 
> this. :-/ In a -Q instance, it turns out that DejaVu Sans Mono 
> (Book) /does/ compose visually:

Can you show a screenshot?

> Composed with the following character(s) "̷" using this font: 
>   xft:-unknown-DejaVu Sans 
>   Mono-normal-normal-normal-*-13-*-*-*-m-0-iso10646-1 
> by these glyphs: 
>   [0 1 111 82 8 0 8 8 1 nil] [0 1 823 703 0 0 8 8 1 [-8 0 0]] 
> 
> Sorry!
> 
> However, Inconsolata-g definitely doesn't compose visually.
> 
> Interestingly, when i use `describe-char' on the 'o', i'm told 
> that the glyph is sourced from Inconsolata-g, but when i use it on 
> the '/', it tells me it's sourced from Gentium ....

Something with your fonts and fontsets, I guess.  If you force Emacs
to use Inconsolata-g for "̷", does the composition happen?

Thanks.

bug#23292: 24.5; Combining characters do not reliably combine

[Alexis]

[-- Attachment #1: Type: text/plain, Size: 1065 bytes --]


Eli Zaretskii <eliz@gnu.org> writes:

> Can you show a screenshot?

Attached.

> Something with your fonts and fontsets, I guess.  If you force 
> Emacs to use Inconsolata-g for "̷", does the composition happen?

Hmm, i'm having some difficulty doing this. i start GUI Emacs from 
an X terminal via 'emacs -Q'. i then go to the Options menu, 
select "Set Default Font", and choose "InconsolataG Medium"[1]. In 
*scratch*, i type 'o', and confirm with `describe-char' that 
"InconsolataG" is the font used for that character.

i then evaluate:

    (set-fontset-font "fontset-default" '(#x0337 . #x0337) 
    "InconsolataG")

with C-x C-e, which returns nil. If i then move point immediately 
after the previously-typed 'o', and type C-x 8 RET 337, the '/' is 
displayed visually separate from the 'o'. But using 
`describe-char' on it says that Gentium is the font used.

i gather something i'm doing something wrong here, i'm guessing in 
the ELisp .... ?

[1] "InconsolataG" is "Inconsolata-g" renamed by me locally, via 
FontForge, so as to be XLFD-friendly.


[-- Attachment #2: dejavu_sans_mono_book.png --]
[-- Type: image/png, Size: 5821 bytes --]

bug#23292: 24.5; Combining characters do not reliably combine

[Eli Zaretskii]

> From: Alexis <flexibeast@gmail.com>
> Cc: handa@gnu.org, hdfssk@gmail.com, 23292@debbugs.gnu.org
> Date: Fri, 15 Apr 2016 21:52:51 +1000
> 
> > Something with your fonts and fontsets, I guess.  If you force 
> > Emacs to use Inconsolata-g for "̷", does the composition happen?
> 
> Hmm, i'm having some difficulty doing this. i start GUI Emacs from 
> an X terminal via 'emacs -Q'. i then go to the Options menu, 
> select "Set Default Font", and choose "InconsolataG Medium"[1]. In 
> *scratch*, i type 'o', and confirm with `describe-char' that 
> "InconsolataG" is the font used for that character.
> 
> i then evaluate:
> 
>     (set-fontset-font "fontset-default" '(#x0337 . #x0337) 
>     "InconsolataG")
> 
> with C-x C-e, which returns nil. If i then move point immediately 
> after the previously-typed 'o', and type C-x 8 RET 337, the '/' is 
> displayed visually separate from the 'o'. But using 
> `describe-char' on it says that Gentium is the font used.

Looks like for some reason Emacs rejects InconsolataG as the font for
that character, not sure why.

bug#23292: 24.5; Combining characters do not reliably combine

[Eli Zaretskii]

> Date: Fri, 15 Apr 2016 11:32:29 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: hdfssk@gmail.com, 23292@debbugs.gnu.org
> 
> For the record, the composition works for me on MS-Windows using the
> Arial Unicode MS font, and the composition data looks quite different
> (and makes much more sense to me) than what the OP shows:
> 
>   Composed with the following character(s) "̷" using this font:
>     uniscribe:-outline-Arial Unicode MS-normal-normal-normal-sans-13-*-*-*-p-*-iso8859-1
>   by these glyphs:
>     [0 1 111 82 7 1 6 14 4 nil]
>     [0 1 823 671 0 -5 -2 14 4 nil]
> 
> The OP said the composition data he gets is this:
> 
>   Composed with the following character(s) "̷" using this font:
>     xft:-PfEd-DejaVu Sans Mono-normal-normal-normal-*-13-*-*-*-m-0-iso10646-1
>   by these glyphs:
>     [0 1 111 82 8 1 7 7 0 nil]
>     [0 1 823 703 8 0 8 8 1 nil]
> 
> and the offsets in the second vector look wrong to me, FWIW.

I now tried this on Windows 8.1, where the (default) Courier New font
does have a glyph for u+0337, and I see there the same problem as
reported by the OP, including the composition data that shows positive
offsets where I thought negative offsets should be.

Maybe this is something related to the fact that bot DejaVu Sans Mono
and Courier New are monospaced fonts, whereas Arial Unicode MS isn't?

I Hope Handa-san will provide some insight.

bug#23292: 24.5; Combining characters do not reliably combine

[handa]

Sorry for the late response.

In article <83y48fcjw9.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> > For the record, the composition works for me on MS-Windows using the
> > Arial Unicode MS font, and the composition data looks quite different
> > (and makes much more sense to me) than what the OP shows:
> > 
> >   Composed with the following character(s) "̷" using this font:
> >     uniscribe:-outline-Arial Unicode MS-normal-normal-normal-sans-13-*-*-*-p-*-iso8859-1
> >   by these glyphs:
> >     [0 1 111 82 7 1 6 14 4 nil]
> >     [0 1 823 671 0 -5 -2 14 4 nil]
> > 
> > The OP said the composition data he gets is this:
> > 
> >   Composed with the following character(s) "̷" using this font:
> >     xft:-PfEd-DejaVu Sans Mono-normal-normal-normal-*-13-*-*-*-m-0-iso10646-1
> >   by these glyphs:
> >     [0 1 111 82 8 1 7 7 0 nil]
> >     [0 1 823 703 8 0 8 8 1 nil]
> > 
> > and the offsets in the second vector look wrong to me, FWIW.

> I now tried this on Windows 8.1, where the (default) Courier New font
> does have a glyph for u+0337, and I see there the same problem as
> reported by the OP, including the composition data that shows positive
> offsets where I thought negative offsets should be.

> Maybe this is something related to the fact that bot DejaVu Sans Mono
> and Courier New are monospaced fonts, whereas Arial Unicode MS isn't?

> I Hope Handa-san will provide some insight.

I tried to display "o\x0337" by "dejavu sans mono" font and saw no
problem.  I also tried "Inconsolata-g" font and found "\x0337" was not
displayed by that font.   But, this is simply because the font doesn't
have a glyph for "\x0337".

I downloaded "Inconsolata-g" font from
http://www.fantascienza.net/leonardo/ar/inconsolatag/inconsolata-g_font.zip.
Are we using the same font?

---
K. Handa
handa@gnu.org

bug#23292: 24.5; Combining characters do not reliably combine

[Alexis]

handa <handa@gnu.org> writes:

> I tried to display "o\x0337" by "dejavu sans mono" font and saw 
> no problem.  I also tried "Inconsolata-g" font and found 
> "\x0337" was not displayed by that font.   But, this is simply 
> because the font doesn't have a glyph for "\x0337".
>
> I downloaded "Inconsolata-g" font from 
> http://www.fantascienza.net/leonardo/ar/inconsolatag/inconsolata-g_font.zip. 
> Are we using the same font?

i believe so.

Running fc-query(1) on the TTF version in my ~/.fonts folder:

    $ fc-query ~/.fonts/ttf/Inconsolata-g.ttf | grep -E 
    'version|hash'  fontversion: 66191(i)(s) hash: 
    "sha256:6847a2f48296546984ea60c37cc34749375857d137a01a574566d9c73d13e908"(s)

Running fc-query(1) on the TTF version from the above URL:

    $ fc-query ./Inconsolata-g.ttf | grep -E 'version|hash' 
    fontversion: 66191(i)(s) hash: 
    "sha256:6847a2f48296546984ea60c37cc34749375857d137a01a574566d9c73d13e908"(s)