bug#35024: 26.1; unicode combining characters displayed incorrectly in emacs-26.1

bug#35024: 26.1; unicode combining characters displayed incorrectly in emacs-26.1

[Mayer Goldberg]

[-- Attachment #1: Type: text/plain, Size: 6950 bytes --]

Hello:

?ש = \u05e9 = "HEBREW LETTER SHIN"
?ָ = \u05b8 = "HEBREW POINT QAMATS"

i.e., "\u05e9\u05b8", should display with the qamats right under the shin,
which is the case under emacs 25 and earlier, but fails in emacs 26: The
qamats is printed below but to the side, and not right under the letter
shin. This happens regardless of the font used, and with professional
OTF fonts that are designed to handle combining characters correctly,
and which are handled correctly by emacs 25 and earlier. This problem is
not unique to Hebrew, but is true for any RTL language I've checked
inclinding Arabic and Farsi.


In GNU Emacs 26.1 (build 3, x86_64-pc-linux-gnu, GTK+ Version 2.24.32)
 of 2019-03-08 built on curry
Windowing system distributor 'The X.Org Foundation', version 11.0.11906000
System Description:    Ubuntu 18.04.2 LTS

Recent messages:
For information about GNU Emacs and the GNU system, type <f1> C-a.
Mark set
Mark saved where search started
Saving file /home/gmayer/brain/main.brain...
Wrote /home/gmayer/brain/main.brain
Making completion list...
Quit [13 times]
Mark set [2 times]
delete-backward-char: Beginning of buffer [2 times]
You can run the command ‘unicode-info’ with M-x un-i RET

Configured using:
 'configure --with-gif=no 'LDFLAGS= -ldl -lm''

Configured features:
XPM JPEG TIFF PNG RSVG SOUND GSETTINGS NOTIFY GNUTLS LIBXML2 FREETYPE
XFT ZLIB TOOLKIT_SCROLL_BARS GTK2 X11 THREADS

Important settings:
  value of $LC_MONETARY: en_US.UTF-8
  value of $LC_NUMERIC: en_US.UTF-8
  value of $LC_TIME: en_US.UTF-8
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=none
  locale-coding-system: utf-8-unix

Major mode: Fundamental

Minor modes in effect:
  diff-auto-refine-mode: t
  shell-dirtrack-mode: t
  global-pretty-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  column-number-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
~/emacs/emacs-fsharp-mode-master/fsharp-mode-completion hides
/home/gmayer/.emacs.d/elpa/fsharp-mode-20190304.1317/fsharp-mode-completion
~/emacs/emacs-fsharp-mode-master/fsharp-mode-indent-smie hides
/home/gmayer/.emacs.d/elpa/fsharp-mode-20190304.1317/fsharp-mode-indent-smie
~/emacs/emacs-fsharp-mode-master/fsharp-doc hides
/home/gmayer/.emacs.d/elpa/fsharp-mode-20190304.1317/fsharp-doc
~/emacs/emacs-fsharp-mode-master/fsharp-mode hides
/home/gmayer/.emacs.d/elpa/fsharp-mode-20190304.1317/fsharp-mode
~/emacs/emacs-fsharp-mode-master/fsharp-mode-indent hides
/home/gmayer/.emacs.d/elpa/fsharp-mode-20190304.1317/fsharp-mode-indent
~/emacs/emacs-fsharp-mode-master/fsharp-mode-font hides
/home/gmayer/.emacs.d/elpa/fsharp-mode-20190304.1317/fsharp-mode-font
~/emacs/emacs-fsharp-mode-master/fsharp-mode-util hides
/home/gmayer/.emacs.d/elpa/fsharp-mode-20190304.1317/fsharp-mode-util
~/emacs/emacs-fsharp-mode-master/flycheck-fsharp hides
/home/gmayer/.emacs.d/elpa/fsharp-mode-20190304.1317/flycheck-fsharp
~/emacs/emacs-fsharp-mode-master/fsharp-mode-pkg hides
/home/gmayer/.emacs.d/elpa/fsharp-mode-20190304.1317/fsharp-mode-pkg
~/emacs/emacs-fsharp-mode-master/inf-fsharp-mode hides
/home/gmayer/.emacs.d/elpa/fsharp-mode-20190304.1317/inf-fsharp-mode
~/emacs/markdown-mode hides
/home/gmayer/.emacs.d/elpa/markdown-mode-20190305.319/markdown-mode
~/emacs/sqlite hides /home/gmayer/.emacs.d/elpa/sqlite-20180708.1711/sqlite
~/emacs/prolog hides /usr/local/share/emacs/26.1/lisp/progmodes/prolog
~/emacs/my-input-methods/greek hides
/usr/local/share/emacs/26.1/lisp/language/greek
~/emacs/my-input-methods/hebrew hides
/usr/local/share/emacs/26.1/lisp/language/hebrew
~/emacs/my-input-methods/tibetan hides
/usr/local/share/emacs/26.1/lisp/language/tibetan

Features:
(shadow sort mail-extr emacsbug message rmc puny dired dired-loaddefs
rfc822 mml mml-sec epa epg gnus-util rmail rmail-loaddefs mm-decode
mm-bodies mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader
sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr mail-utils
misearch multi-isearch elec-pair vc-git diff-mode brain
brain-basic-lisp-functions proof-site proof-autoloads pg-vars prolog
align info htmlize fsharp-mode fsharp-mode-indent-smie smie compile
inf-fsharp-mode fsharp-doc flycheck-fsharp flycheck cl-extra
fsharp-mode-completion fsharp-mode-util etags xref project company pcase
dash subr-x s tramp tramp-compat tramp-loaddefs trampver ucs-normalize
shell parse-time ox-beamer ox-odt rng-loc rng-uri rng-parse rng-match
rng-dt rng-util rng-pttrn nxml-parse nxml-ns nxml-enc xmltok nxml-util
ox-latex ox-icalendar ox-html table ox-ascii ox-publish ox org-element
avl-tree generator org advice org-macro org-footnote org-pcomplete
pcomplete org-list org-faces org-entities noutline outline org-version
ob-emacs-lisp ob ob-tangle org-src ob-ref ob-lob ob-table ob-keys
ob-comint comint ansi-color ring org-loaddefs format-spec find-func
cal-menu calendar cal-loaddefs ob-exp ob-core org-compat ob-eval
org-macs org-install cobol-mode skeleton pretty-mode easy-mmode cl aes
time-stamp quail help-mode 2048-game nasm-mode derived imenu edmacro
kmacro rust-mode rx json map finder-inf package easymenu epg-config
url-handlers url-parse auth-source cl-seq eieio eieio-core cl-macs
eieio-loaddefs password-cache url-vars seq byte-opt gv bytecomp
byte-compile cconv cl-loaddefs cl-lib time-date mule-util tooltip eldoc
electric uniquify ediff-hook vc-hooks lisp-float-type mwheel term/x-win
x-win term/common-win x-dnd tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode elisp-mode lisp-mode
prog-mode register page menu-bar rfn-eshadow isearch timer select
scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame cl-generic cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932
hebrew greek romanian slovak czech european ethiopic indian cyrillic
chinese composite charscript charprop case-table epa-hook jka-cmpr-hook
help simple abbrev obarray minibuffer cl-preloaded nadvice loaddefs
button faces cus-face macroexp files text-properties overlay sha1 md5
base64 format env code-pages mule custom widget hashtable-print-readable
backquote inotify dynamic-setting system-font-setting
font-render-setting move-toolbar gtk x-toolkit x multi-tty
make-network-process emacs)

Memory information:
((conses 16 404821 20970)
 (symbols 48 47007 5)
 (miscs 40 101 241)
 (strings 32 103962 4290)
 (string-bytes 1 2912090)
 (vectors 16 65717)
 (vector-slots 8 1794428 206870)
 (floats 8 142 155)
 (intervals 56 485 0)
 (buffers 992 15))

[-- Attachment #2: Type: text/html, Size: 7367 bytes --]

bug#35024: 26.1; unicode combining characters displayed incorrectly in emacs-26.1

[Basil L. Contovounesios]

[-- Attachment #1: qamats.png --]
[-- Type: image/png, Size: 135838 bytes --]

[-- Attachment #2: Type: text/plain, Size: 3222 bytes --]


Mayer Goldberg <gmayer@little-lisper.org> writes:

> Hello:
>
> ?ש = \u05e9 = "HEBREW LETTER SHIN"
> ?ָ = \u05b8 = "HEBREW POINT QAMATS"
>
> i.e., "\u05e9\u05b8", should display with the qamats right under the shin,
> which is the case under emacs 25 and earlier, but fails in emacs 26: The
> qamats is printed below but to the side, and not right under the letter
> shin. This happens regardless of the font used, and with professional
> OTF fonts that are designed to handle combining characters correctly,
> and which are handled correctly by emacs 25 and earlier. This problem is
> not unique to Hebrew, but is true for any RTL language I've checked
> inclinding Arabic and Farsi.

FWIW, I don't see the behaviour you describe from 'emacs -Q' using a
checkout of the emacs-26 branch (what will be Emacs 26.2):

0. emacs -Q
1. M-: (insert "\u05e9\u05b8") RET
2. C-b C-u C-x =

This is what what-cursor-position reports:

--8<---------------cut here---------------start------------->8---
             position: 146 of 147 (99%), column: 0
            character: ש‎ (displayed as ש‎) (codepoint 1513, #o2751, #x5e9)
              charset: unicode (Unicode (ISO10646))
code point in charset: 0x05E9
               script: hebrew
               syntax: w 	which means: word
             category: .:Base, R:Right-to-left (strong)
             to input: type "C-x 8 RET 5e9" or "C-x 8 RET HEBREW LETTER SHIN"
          buffer code: #xD7 #xA9
            file code: #xD7 #xA9 (encoded by coding system utf-8-unix)
              display: composed to form "שָ" (see below)

Composed with the following character(s) "ָ" using this font:
  xft:-PfEd-Linux Libertine Display O-normal-normal-normal-*-15-*-*-*-*-0-iso10646-1
by these glyphs:
  [0 1 1513 1005 9 0 9 9 0 nil]
  [0 1 1464 967 0 -7 -4 -1 4 [-8 0 0]]

Character code properties: customize what to show
  name: HEBREW LETTER SHIN
  general-category: Lo (Letter, Other)
  decomposition: (1513) ('ש')

There are text properties here:
  fontified            t
--8<---------------cut here---------------end--------------->8---

And attached is a screenshot of how it looks on my screen (scaled with
repeated calls to text-scale-adjust).

-- 
Basil

In GNU Emacs 26.1.92 (build 20, x86_64-pc-linux-gnu, X toolkit, Xaw3d scroll bars)
 of 2019-03-21 built on thunk
Repository revision: 0924b27bca40d219e34529144ea04a581428f1f7
Windowing system distributor 'The X.Org Foundation', version 11.0.12003000
System Description:	Debian GNU/Linux buster/sid

Configured using:
 'configure 'CC=ccache gcc' 'CFLAGS=-O0 -g3 -ggdb -gdwarf-4 -pipe'
 --config-cache --prefix=/home/blc/.local --program-suffix=26
 --enable-checking=yes,glyphs --enable-check-lisp-object-type
 --with-mailutils --with-x-toolkit=lucid --with-modules
 --with-file-notification=yes --with-x'

Configured features:
XAW3D XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GSETTINGS
GLIB NOTIFY ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT
ZLIB TOOLKIT_SCROLL_BARS LUCID X11 XDBE XIM MODULES THREADS LIBSYSTEMD
LCMS2

Important settings:
  value of $LANG: en_IE.UTF-8
  locale-coding-system: utf-8-unix

bug#35024: 26.1; unicode combining characters displayed incorrectly in emacs-26.1

[Eli Zaretskii]

> From: "Basil L. Contovounesios" <contovob@tcd.ie>
> Date: Thu, 28 Mar 2019 00:17:42 +0000
> Cc: 35024@debbugs.gnu.org
> 
> > Hello:
> >
> > ?ש = \u05e9 = "HEBREW LETTER SHIN"
> > ?ָ = \u05b8 = "HEBREW POINT QAMATS"
> >
> > i.e., "\u05e9\u05b8", should display with the qamats right under the shin,
> > which is the case under emacs 25 and earlier, but fails in emacs 26: The
> > qamats is printed below but to the side, and not right under the letter
> > shin. This happens regardless of the font used, and with professional
> > OTF fonts that are designed to handle combining characters correctly,
> > and which are handled correctly by emacs 25 and earlier. This problem is
> > not unique to Hebrew, but is true for any RTL language I've checked
> > inclinding Arabic and Farsi.
> 
> FWIW, I don't see the behaviour you describe from 'emacs -Q' using a
> checkout of the emacs-26 branch (what will be Emacs 26.2):

Right, and neither do I.

I think the reason why Mayer doesn't see correct shaping of
diacriticals is that his Emacs is built without libotf and m17n-flt:

> Configured features:
> XPM JPEG TIFF PNG RSVG SOUND GSETTINGS NOTIFY GNUTLS LIBXML2 FREETYPE
> XFT ZLIB TOOLKIT_SCROLL_BARS GTK2 X11 THREADS

Whereas yours, Basil, does have those libraries:

> Configured features:
> XAW3D XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GSETTINGS
> GLIB NOTIFY ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT
> ZLIB TOOLKIT_SCROLL_BARS LUCID X11 XDBE XIM MODULES THREADS LIBSYSTEMD
> LCMS2

These two libraries are required for complex script shaping, see
README.  Crystal ball says Mayer's Emacs 25 was linked against those
two libraries, that's why it works there.

bug#35024: 26.1; unicode combining characters displayed incorrectly in emacs-26.1

[Mayer Goldberg]

[-- Attachment #1: Type: text/plain, Size: 2198 bytes --]

Dear Eli, Basil:

I used your configuration features to indicate which libraries I need to
install (added much more than just m17n and libotf), re-configured, and
re-built, and now everything is fine. Thank you very much! I've learned a
lot from this process. Thanks for all your help!

Mayer

On Thu, Mar 28, 2019 at 6:12 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: "Basil L. Contovounesios" <contovob@tcd.ie>
> > Date: Thu, 28 Mar 2019 00:17:42 +0000
> > Cc: 35024@debbugs.gnu.org
> >
> > > Hello:
> > >
> > > ?ש = \u05e9 = "HEBREW LETTER SHIN"
> > > ?ָ = \u05b8 = "HEBREW POINT QAMATS"
> > >
> > > i.e., "\u05e9\u05b8", should display with the qamats right under the
> shin,
> > > which is the case under emacs 25 and earlier, but fails in emacs 26:
> The
> > > qamats is printed below but to the side, and not right under the letter
> > > shin. This happens regardless of the font used, and with professional
> > > OTF fonts that are designed to handle combining characters correctly,
> > > and which are handled correctly by emacs 25 and earlier. This problem
> is
> > > not unique to Hebrew, but is true for any RTL language I've checked
> > > inclinding Arabic and Farsi.
> >
> > FWIW, I don't see the behaviour you describe from 'emacs -Q' using a
> > checkout of the emacs-26 branch (what will be Emacs 26.2):
>
> Right, and neither do I.
>
> I think the reason why Mayer doesn't see correct shaping of
> diacriticals is that his Emacs is built without libotf and m17n-flt:
>
> > Configured features:
> > XPM JPEG TIFF PNG RSVG SOUND GSETTINGS NOTIFY GNUTLS LIBXML2 FREETYPE
> > XFT ZLIB TOOLKIT_SCROLL_BARS GTK2 X11 THREADS
>
> Whereas yours, Basil, does have those libraries:
>
> > Configured features:
> > XAW3D XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GSETTINGS
> > GLIB NOTIFY ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT
> > ZLIB TOOLKIT_SCROLL_BARS LUCID X11 XDBE XIM MODULES THREADS LIBSYSTEMD
> > LCMS2
>
> These two libraries are required for complex script shaping, see
> README.  Crystal ball says Mayer's Emacs 25 was linked against those
> two libraries, that's why it works there.
>

[-- Attachment #2: Type: text/html, Size: 2914 bytes --]

bug#35024: 26.1; unicode combining characters displayed incorrectly in emacs-26.1

[Eli Zaretskii]

> From: Mayer Goldberg <gmayer@little-lisper.org>
> Date: Thu, 28 Mar 2019 19:15:10 +0200
> Cc: "Basil L. Contovounesios" <contovob@tcd.ie>, 35024@debbugs.gnu.org
> 
> Dear Eli, Basil:
> 
> I used your configuration features to indicate which libraries I need to install (added much more than just m17n
> and libotf), re-configured, and re-built, and now everything is fine. Thank you very much! I've learned a lot from
> this process. Thanks for all your help!

Thanks, I'm therefore closing the bug.