bug#21846: 25.0.50; emacs hebrew script bidi nikud rendering bug

bug#21846: 25.0.50; emacs hebrew script bidi nikud rendering bug

[Mark H. David]

emacs -Q
C-x C-f ytest-emacs.txt RET

This was using the default font and no other changes or customizations.

You see numerous glaring rendering bugs.

First case:
  Upper right first sequence (bounded by space)
  Sequence description: mem double-yud pasekh sof-nun
  Paste of character sequence: מײַן
  Problem: a "base character standin" glyph appears between what should
  be the 2nd/3rd of 3 bounding boxes, the second of which comprises two
  Unicode characters:
    05F2;HEBREW LIGATURE YIDDISH DOUBLE YOD
    05B7;HEBREW POINT PATAH

Second case: 
  Upper right 5th sequence (bounded by space)
  Sequence description: alef pasekh pey rofe reysh yud ayin
  Paste of character sequence:  אַפֿריקע
  Problem: the base character alef and pey are separated from
    their following nikuds


Many more could be described. I'll stop at two for now.  Should they all
be combined, or should they all be divided?

I could send screen shots, both of how it should look and how it
actually looks, but how do I do that? I don't see a way with this bug
report. Can you give instructions on how to do that if it's useful?

Thanks,

-Mark





In GNU Emacs 25.0.50.1 (x86_64-unknown-linux-gnu, GTK+ Version 3.10.8)
 of 2015-11-05
Repository revision: 54e2ed97bf904eda2ee5ae9b7822288e10b3abc5
Windowing system distributor 'The X.Org Foundation', version 11.0.11701000
System Description:	Ubuntu 14.04.3 LTS

Configured features:
XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GCONF GSETTINGS
NOTIFY LIBSELINUX GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB
TOOLKIT_SCROLL_BARS GTK3 X11

Important settings:
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=ibus
  locale-coding-system: utf-8-unix

Major mode: Text

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Recent messages:
For information about GNU Emacs and the GNU system, type C-h C-a.

Load-path shadows:
None found.

Features:
(shadow sort gnus-util mail-extr emacsbug message dired format-spec
rfc822 mml mml-sec mm-decode mm-bodies mm-encode mail-parse rfc2231
mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045 ietf-drums
mm-util help-fns help-mode easymenu cl-loaddefs pcase cl-lib mail-prsvr
mail-utils time-date mule-util tooltip eldoc electric uniquify
ediff-hook vc-hooks lisp-float-type mwheel x-win term/common-win x-dnd
tool-bar dnd fontset image regexp-opt fringe tabulated-list newcomment
elisp-mode lisp-mode prog-mode register page menu-bar rfn-eshadow timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
frame cl-generic cham georgian utf-8-lang misc-lang vietnamese tibetan
thai tai-viet lao korean japanese eucjp-ms cp51932 hebrew greek romanian
slovak czech european ethiopic indian cyrillic chinese charscript
case-table epa-hook jka-cmpr-hook help simple abbrev minibuffer
cl-preloaded nadvice loaddefs button faces cus-face macroexp files
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget hashtable-print-readable backquote dbusbind inotify
dynamic-setting system-font-setting font-render-setting move-toolbar gtk
x-toolkit x multi-tty make-network-process emacs)

Memory information:
((conses 16 81228 5903)
 (symbols 48 19102 0)
 (miscs 40 287 135)
 (strings 32 12876 4236)
 (string-bytes 1 378178)
 (vectors 16 15082)
 (vector-slots 8 458413 6458)
 (floats 8 133 86)
 (intervals 56 192 0)
 (buffers 976 12)
 (heap 1024 30041 988))

bug#21846: Additional Info re: bug#21846 (25.0.50; emacs hebrew script bidi nikud rendering bug)

[Mark David]

[-- Attachment #1: Type: text/plain, Size: 415 bytes --]

Re:
  http://debbugs.gnu.org/cgi/bugreport.cgi?bug=21846

See attachments.

font: Droid Sans Mono, Size: 10
screen shot:
  ytest-emacs-Screenshot_2015-11-06_14.10.12.png
file opened:
  ytest-emacs.txt
file showing how example problem should look rendered in browser (firefox/linux):
  ytest-how-should-look-rendered-ok_Screenshot_2015-11-06_16.03.29.png
URL for above:
  http://uyip.org/ytest/mashinke-linux.html



[-- Attachment #2: ytest-how-should-look-rendered-ok_Screenshot_2015-11-06_16.03.29.png --]
[-- Type: image/png, Size: 150687 bytes --]

[-- Attachment #3: ytest-emacs.txt --]
[-- Type: text/plain, Size: 3733 bytes --]


מײַן שײן הױז אין אַפֿריקע קוקט װוּהין סע פּרוּװט; רויִק און אָרעם פֿליִען פֿריִיִקע פֿעיִקע בליאַסקן; ייִדן לאַכן, פּאַטשן, טאַנצן, און פּלאַצן מיט גוטס. װען־װען איך; איך שטאָף? שטאָף קרעם! קרעם אַלץ: אַלץ. מידקײט קײַט אױף דער הױט מיט פֿרײַהײט און מילדקײטן פֿון פֿרײַהײטס װעגן. שױן ,,זשומעט'' אַזש די דזשוקן אױסהערן נ. מ. װאַרשאַװסקין. אױ, אַן אײ, אײַ אײַ אײַ! צװישן 12:40 און 12־50, נאָכמיטאָג. מיניִיאַטורן י'ן טוּונג אן נא דער פֿאַרחלומטער אײדעם אַבֿרהם פֿון שׂונאימלאַנד האָט פֿון תּמיד־אָן אַזױ געזאָגט. די פֿאַראײניקטע שטאַטן פֿאַראײניקן אין אַ פֿאַרײַן פֿון שטעט. ‬ 

אבגדהוזחטיךכלםמןנסעףפץצקרשת|װ|ױ|ײ|אָ|אַ|בֿ|וּ|יִ|ײַ|כּ|פּ|פֿ|שׂ|תּ 

 א אְ אֱ אֲ אֳ אִ אֵ אֶ אַ אָ
ב בְ בֱ בֲ בֳ בִ בֵ בֶ בַ בָ
בּ בְּ בֱּ בֲּ בֳּ בִּ בֵּ בֶּ בַּ בָּ
×’ ×’Ö° ×’Ö± ×’Ö² ×’Ö³ ×’Ö´ ×’Öµ ×’Ö¶ ×’Ö· ×’Ö¸
×’Ö¼ ×’Ö°Ö¼ ×’Ö±Ö¼ ×’Ö²Ö¼ ×’Ö³Ö¼ ×’Ö´Ö¼ ×’ÖµÖ¼ ×’Ö¶Ö¼ ×’Ö·Ö¼ ×’Ö¸Ö¼
ד דְ דֱ דֲ דֳ דִ דֵ דֶ דַ דָ
דּ דְּ דֱּ דֲּ דֳּ דִּ דֵּ דֶּ דַּ דָּ
×” ×”Ö° ×”Ö± ×”Ö² ×”Ö³ ×”Ö´ ×”Öµ ×”Ö¶ ×”Ö· ×”Ö¸
ו וְ וֱ וֲ וֳ וִ וֵ וֶ וַ וָ
וּ וְּ וֱּ וֲּ וֳּ וִּ וֵּ וֶּ וַּ וָּ
×° ×°Ö· ×°Ö¸ ×°Öµ ×°Ö´ ×°Ö¶ ×°Ö°
×±
×– ×–Ö° ×–Ö± ×–Ö² ×–Ö³ ×–Ö´ ×–Öµ ×–Ö¶ ×–Ö· ×–Ö¸
×–Ö¼ ×–Ö°Ö¼ ×–Ö±Ö¼ ×–Ö²Ö¼ ×–Ö³Ö¼ ×–Ö´Ö¼ ×–ÖµÖ¼ ×–Ö¶Ö¼ ×–Ö·Ö¼ ×–Ö¸Ö¼
×— ×—Ö° ×—Ö± ×—Ö² ×—Ö³ ×—Ö´ ×—Öµ ×—Ö¶ ×—Ö· ×—Ö¸
ט טְ טֱ טֲ טֳ טִ טֵ טֶ טַ טָ
טּ טְּ טֱּ טֲּ טֳּ טִּ טֵּ טֶּ טַּ טָּ
×™ ×™Ö° ×™Ö± ×™Ö² ×™Ö³ ×™Ö´ ×™Öµ ×™Ö¶ ×™Ö· ×™Ö¸
×™Ö¼ ×™Ö°Ö¼ ×™Ö±Ö¼ ×™Ö²Ö¼ ×™Ö³Ö¼ ×™Ö´Ö¼ ×™ÖµÖ¼ ×™Ö¶Ö¼ ×™Ö·Ö¼ ×™Ö¸Ö¼
ײ ײַ ײֵ ײָ
×› ×›Ö° ×›Ö± ×›Ö² ×›Ö³ ×›Ö´ ×›Öµ ×›Ö¶ ×›Ö· ×›Ö¸
×›Ö¼ ×›Ö°Ö¼ ×›Ö±Ö¼ ×›Ö²Ö¼ ×›Ö³Ö¼ ×›Ö´Ö¼ ×›ÖµÖ¼ ×›Ö¶Ö¼ ×›Ö·Ö¼ ×›Ö¸Ö¼
ל לְ לֱ לֲ לֳ לִ לֵ לֶ לַ לָ
לּ לְּ לֱּ לֲּ לֳּ לִּ לֵּ לֶּ לַּ לָּ
מ מְ מֱ מֲ מֳ מִ מֵ מֶ מַ מָ
מּ מְּ מֱּ מֲּ מֳּ מִּ מֵּ מֶּ מַּ מָּ
×  × Ö° × Ö± × Ö² × Ö³ × Ö´ × Öµ × Ö¶ × Ö· × Ö¸
× Ö¼ × Ö°Ö¼ × Ö±Ö¼ × Ö²Ö¼ × Ö³Ö¼ × Ö´Ö¼ × ÖµÖ¼ × Ö¶Ö¼ × Ö·Ö¼ × Ö¸Ö¼
ס סְ סֱ סֲ סֳ סִ סֵ סֶ סַ סָ
סּ סְּ סֱּ סֲּ סֳּ סִּ סֵּ סֶּ סַּ סָּ
×¢ ×¢Ö° ×¢Ö± ×¢Ö² ×¢Ö³ ×¢Ö´ ×¢Öµ ×¢Ö¶ ×¢Ö· ×¢Ö¸
פ פְ פֱ פֲ פֳ פִ פֵ פֶ פַ פָ
פּ פְּ פֱּ פֲּ פֳּ פִּ פֵּ פֶּ פַּ פָּ
צ צְ צֱ צֲ צֳ צִ צֵ צֶ צַ צָ
צּ צְּ צֱּ צֲּ צֳּ צִּ צֵּ צֶּ צַּ צָּ
ק קְ קֱ קֲ קֳ קִ קֵ קֶ קַ קָ
קּ קְּ קֱּ קֲּ קֳּ קִּ קֵּ קֶּ קַּ קָּ
ר רְ רֱ רֲ רֳ רִ רֵ רֶ רַ רָ
ש שְ שֱ שֲ שֳ שִ שֵ שֶ שַ שָ
שּ שְּ שֱּ שֲּ שֳּ שִּ שֵּ שֶּ שַּ שָּ
שׂ שְׂ שֱׂ שֲׂ שֳׂ שִׂ שֵׂ שֶׂ שַׂ שָׂ
שּׂ שְּׂ שֱּׂ שֲּׂ שֳּׂ שִּׂ שֵּׂ שֶּׂ שַּׂ שָּׂ
שׁ שְׁ שֱׁ שֲׁ שֳׁ שִׁ שֵׁ שֶׁ שַׁ שָׁ
שּׁ שְּׁ שֱּׁ שֲּׁ שֳּׁ שִּׁ שֵּׁ שֶּׁ שַּׁ שָּׁ
ת תְ תֱ תֲ תֳ תִ תֵ תֶ תַ תָ
תּ תְּ תֱּ תֲּ תֳּ תִּ תֵּ תֶּ תַּ תָּ











[-- Attachment #4: ytest-emacs-Screenshot_2015-11-06_14.10.12.png --]
[-- Type: image/png, Size: 101507 bytes --]

bug#21846: 25.0.50; emacs hebrew script bidi nikud rendering bug

[Eli Zaretskii]

[-- Attachment #1: Type: text/plain, Size: 2090 bytes --]

> From: mhd@yv.org (Mark H. David)
> Date: Fri, 06 Nov 2015 14:29:43 -0800
> 
> emacs -Q
> C-x C-f ytest-emacs.txt RET
> 
> This was using the default font and no other changes or customizations.
> 
> You see numerous glaring rendering bugs.
> 
> First case:
>   Upper right first sequence (bounded by space)
>   Sequence description: mem double-yud pasekh sof-nun
>   Paste of character sequence: מײַן
>   Problem: a "base character standin" glyph appears between what should
>   be the 2nd/3rd of 3 bounding boxes, the second of which comprises two
>   Unicode characters:
>     05F2;HEBREW LIGATURE YIDDISH DOUBLE YOD
>     05B7;HEBREW POINT PATAH
> 
> Second case: 
>   Upper right 5th sequence (bounded by space)
>   Sequence description: alef pasekh pey rofe reysh yud ayin
>   Paste of character sequence:  אַפֿריקע
>   Problem: the base character alef and pey are separated from
>     their following nikuds
> 
> 
> Many more could be described. I'll stop at two for now.  Should they all
> be combined, or should they all be divided?

See the attached image from my system (which is not Ubuntu).  As you
see, the same version of Emacs displays the Hebrew diacriticals quite
correctly.

So the problem is likely in the shaping engine used to build Emacs.
I'm CC'ing Handa-san who is our expert on complex text layout support.
What is the version of libotf, m17n-db, and libm17n-flt libraries used
by Emacs on your system?  Perhaps upgrading to the latest versions of
those libraries will fix the problem.

> I could send screen shots, both of how it should look and how it
> actually looks, but how do I do that? I don't see a way with this bug
> report. Can you give instructions on how to do that if it's useful?

I see you've sent the screenshots, thanks.  What you show is clear:
the complex test layout doesn't work correctly.  Since it does on my
system, with a different shaping engine, I'm quite sure the reason is
outside Emacs.

Here's a screenshot from my system displaying the same ytest-emacs.txt
file:


[-- Attachment #2: ytest-emacs-on-Eli's-machine.png --]
[-- Type: image/png, Size: 27644 bytes --]

bug#21846: 25.0.50; emacs hebrew script bidi nikud rendering bug

[Mark David]

Does the following give the answer?


On 11/7/15, 12:59 AM, "Eli Zaretskii" <eliz@gnu.org> wrote:

>So the problem is likely in the shaping engine used to build Emacs.
>I'm CC'ing Handa-san who is our expert on complex text layout support.
>What is the version of libotf, m17n-db, and libm17n-flt libraries used
>by Emacs on your system?  Perhaps upgrading to the latest versions of
>those libraries will fix the problem.

mhd:/home/mhd/emacs$ dpkg -l "*libotf*"
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                      Version           Architecture      Description
+++-=========================-=================-=================-========================================================
ii  libotf-dev:amd64          0.9.13-1ubuntu1   amd64             Library for handling OpenType Font - development
ii  libotf0:amd64             0.9.13-1ubuntu1   amd64             Library for handling OpenType Font - runtime
mhd:/home/mhd/emacs$ dpkg -l "*m17*"
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                      Version           Architecture      Description
+++-=========================-=================-=================-========================================================
ii  libm17n-0                 1.6.4-2ubuntu1    amd64             multilingual text processing library - runtime
ii  libm17n-dev               1.6.4-2ubuntu1    amd64             multilingual text processing library - development
ii  m17n-contrib              1.1.14-1          all               multilingual text processing library - contributed datab
ii  m17n-db                   1.6.4-1           all               multilingual text processing library - database
ii  m17n-docs                 1.6.2-2           all               multilingual text processing library - documents



If not, can someone give hints how to supply the desired info?
Also hints please on how to upgrade to the latest versions?

Thanks,

Mark

bug#21846: 25.0.50; emacs hebrew script bidi nikud rendering bug

[Benjamin Riefenstahl]

[-- Attachment #1: Type: text/plain, Size: 845 bytes --]

I can reproduce the problems with the default font (FreeMono) on my
system, which for m17n has a mixture of Versions 1.6.4 and 1.6.5.  This
is Debian GNU/Linux 8.2.

When I use "FreeSerif", a small part of the problems go away, see the
attached screenshots.

OTOH it seems that the other problems will need to be be fixed in m17n.
E.g. I can fix the first problem (yiddish + nikud) by making a
relatively simple change to m17n's HEBR-OTF.flt script.  It may even be
possible to fix the problems with fonts like "FreeMono" and "Droid Sans
Mono" using m17n.

As for m17n versions, the file HEBR-OTF.flt has not changed since it was
initially checked-in to CVS (it currently lives in Git in
<http://git.savannah.gnu.org/cgit/m17n/m17n-db.git/>).

I suggest we move this bug to the m17n bugtracker at
<http://savannah.nongnu.org/bugs/?group=m17n>.


[-- Attachment #2: page1-freeserif.png --]
[-- Type: image/png, Size: 109023 bytes --]

[-- Attachment #3: page2-freeserif.png --]
[-- Type: image/png, Size: 55741 bytes --]

bug#21846: 25.0.50; emacs hebrew script bidi nikud rendering bug

[Eli Zaretskii]

> From: Benjamin Riefenstahl <b.riefenstahl@turtle-trading.net>
> Cc: Eli Zaretskii <eliz@gnu.org>,
>     Mark H. David <mhd@yv.org>,
>     Kenichi Handa <handa@gnu.org>
> Date: Sat, 07 Nov 2015 18:44:14 +0100
> 
> I can reproduce the problems with the default font (FreeMono) on my
> system, which for m17n has a mixture of Versions 1.6.4 and 1.6.5.  This
> is Debian GNU/Linux 8.2.

I believe these are not the latest versions of these libraries.

> When I use "FreeSerif", a small part of the problems go away, see the
> attached screenshots.

Emacs cannot compose characters unless they come from the same font.
Are you sure the base characters and the diacriticals come from the
same font, in those cases where the display is incorrect?

> I suggest we move this bug to the m17n bugtracker at
> <http://savannah.nongnu.org/bugs/?group=m17n>.

A good idea, IMO.

bug#21846: 25.0.50; emacs hebrew script bidi nikud rendering bug

[Benjamin Riefenstahl]

Eli Zaretskii writes:

>> From: Benjamin Riefenstahl <b.riefenstahl@turtle-trading.net>
>> When I use "FreeSerif", a small part of the problems go away, see the
>> attached screenshots.
>
> Emacs cannot compose characters unless they come from the same font.
> Are you sure the base characters and the diacriticals come from the
> same font, in those cases where the display is incorrect?

Right.  I checked that now.  For me the default configuration (emacs -Q)
uses "DejaVu Sans Mono" for numbers and spaces and "FreeMono" for
Hebrew, base characters as well as nikud.  After I call M-x
set-default-font RET FreeSerif RET, that font is used for all
characters.

@Mark, can you check that in your case "Droid Sans Mono" is actually
used for the Hebrew?  Put the cursor on some on the character in
question and do M-x describe-char RET, the font should be mentioned in
the resulting help text.

>> I suggest we move this bug to the m17n bugtracker at
>> <http://savannah.nongnu.org/bugs/?group=m17n>.
>
> A good idea, IMO.

Done, see <https://savannah.nongnu.org/bugs/index.php?46399>.  I put all
current correspondents on CC there, I hope that's o.k.  Otherwise I can
remove any of you again, I guess.  ;-)

I'll add what I found so far there.

bug#21846: 25.0.50; emacs hebrew script bidi nikud rendering bug

[Mark David]

Requested info below:

On 11/7/15, 11:24 AM, "Benjamin Riefenstahl" <b.riefenstahl@turtle-trading.net> wrote:




>Eli Zaretskii writes:
>
>>>From: Benjamin Riefenstahl <b.riefenstahl@turtle-trading.net>
>>>When I use "FreeSerif", a small part of the problems go away, see the
>>>attached screenshots.
>>
>>Emacs cannot compose characters unless they come from the same font.
>>Are you sure the base characters and the diacriticals come from the
>>same font, in those cases where the display is incorrect?
>
>Right.  I checked that now.  For me the default configuration (emacs -Q)
>uses "DejaVu Sans Mono" for numbers and spaces and "FreeMono" for
>Hebrew, base characters as well as nikud.  After I call M-x
>set-default-font RET FreeSerif RET, that font is used for all
>characters.
>
>@Mark, can you check that in your case "Droid Sans Mono" is actually
>used for the Hebrew?  Put the cursor on some on the character in
>question and do M-x describe-char RET, the font should be mentioned in
>the resulting help text.

With the cursor on the double yud of the first "paragraph" of the original ytest.txt:

             position: 8 of 2166 (0%), column: 5
character: ײ‎ (displayed as ײ‎) (codepoint 1522, #o2762, #x5f2)
preferred charset: unicode (Unicode (ISO10646))
code point in charset: 0x05F2
script: hebrew
syntax: w which means: word
category: .:Base, R:Right-to-left (strong)
to input: type "C-x 8 RET 5f2" or "C-x 8 RET HEBREW LIGATURE YIDDISH DOUBLE YOD"
buffer code: #xD7 #xB2
file code: #xD7 #xB2 (encoded by coding system utf-8-unix)
display: by this font (glyph code)
xft:-unknown-FreeMono-normal-normal-normal-*-18-*-*-*-m-0-iso10646-1 (#x504)

Character code properties: customize what to show
name: HEBREW LIGATURE YIDDISH DOUBLE YOD
old-name: HEBREW LETTER DOUBLE YOD
general-category: Lo (Letter, Other)
decomposition: (1522) ('ײ')


With cursor on the daled in the second "paragraph" of the original ytest.txt:

             position: 580 of 2166 (27%), column: 3
character: ד‎ (displayed as ד‎) (codepoint 1491, #o2723, #x5d3)
preferred charset: iso-8859-8 (ISO/IEC 8859/8)
code point in charset: 0xE3
script: hebrew
syntax: w which means: word
category: .:Base, R:Right-to-left (strong)
to input: type "C-x 8 RET 5d3" or "C-x 8 RET HEBREW LETTER DALET"
buffer code: #xD7 #x93
file code: #xD7 #x93 (encoded by coding system utf-8-unix)
display: by this font (glyph code)
xft:-unknown-FreeMono-normal-normal-normal-*-18-*-*-*-m-0-iso10646-1 (#x4EA)

Character code properties: customize what to show
name: HEBREW LETTER DALET
general-category: Lo (Letter, Other)
decomposition: (1491) ('ד')

bug#21846: 25.0.50; emacs hebrew script bidi nikud rendering bug

[Benjamin Riefenstahl]

Mark David writes:
> character: ײ‎ (displayed as ײ‎) (codepoint 1522, #o2762, #x5f2)
> [...]
> xft:-unknown-FreeMono-normal-normal-normal-*-18-*-*-*-m-0-iso10646-1 (#x504)
>
> [...]
> character: ד‎ (displayed as ד‎) (codepoint 1491, #o2723, #x5d3)
> [...]
> xft:-unknown-FreeMono-normal-normal-normal-*-18-*-*-*-m-0-iso10646-1 (#x4EA)

Thanks.  So in your case also, Emacs falls back to "FreeMono".  That
eliminates "Droid Sans Mono" from the picture, I presume because it does
not have Hebrew at all.

bug#21846: 25.0.50; emacs hebrew script bidi nikud rendering bug

[Benjamin Riefenstahl]

All the problems with the test text that I can see are fixed in m17n,
see the report opened there
<https://savannah.nongnu.org/bugs/index.php?46399>.

@Mark: Can you check that there is nothing open from your POV?

Workaround procedure: As long as there is not new version of m17n
(i.e. >= 1.7.0) that can just be installed through package management,
the fixes so far can be applied by creating a directory ~/.m17n.d and
putting the file
<http://git.savannah.gnu.org/cgit/m17n/m17n-db.git/plain/FLT/HEBR-OTF.flt>
there.

bug#21846: 25.0.50; emacs hebrew script bidi nikud rendering bug

[Mark David]

[-- Attachment #1: Type: text/plain, Size: 2149 bytes --]

The nikud placement is good to excellent in the fonts tried so far.  

However, I tried a few more, and found one that's terrible: Aharoni

Following is output from M-x describe-char with cursor on the double-yud character in the first word of the first paragraph.  This font seems to systematically misplace not all but most base char + nikud combinations in my sample file.  (Screen shot attached.)  Any way to fix this?  

Thanks,

Mark

             position: 3 of 2166 (0%), column: 1
            character: ײ‎ (displayed as ײ‎) (codepoint 1522, #o2762, #x5f2)
    preferred charset: unicode (Unicode (ISO10646))
code point in charset: 0x05F2
               script: hebrew
               syntax: w 	which means: word
             category: .:Base, R:Right-to-left (strong)
             to input: type "C-x 8 RET 5f2" or "C-x 8 RET HEBREW LIGATURE YIDDISH DOUBLE YOD"
          buffer code: #xD7 #xB2
            file code: #xD7 #xB2 (encoded by coding system utf-8-unix)
              display: composed to form "ײַ" (see below)

Composed with the following character(s) "ַ" using this font:
  xft:-culmus-Aharoni CLM-semilight-normal-normal-*-33-*-*-*-*-0-iso10646-1
by these glyphs:
  [0 1 1522 145 13 -1 12 18 -10 nil]
  [0 1 1463 103 0 -5 5 -2 4 nil]

Character code properties: customize what to show
  name: HEBREW LIGATURE YIDDISH DOUBLE YOD
  old-name: HEBREW LETTER DOUBLE YOD
  general-category: Lo (Letter, Other)
  decomposition: (1522) ('ײ')






On 11/28/15, 2:33 PM, "Benjamin Riefenstahl" <b.riefenstahl@turtle-trading.net> wrote:

>All the problems with the test text that I can see are fixed in m17n,
>see the report opened there
><https://savannah.nongnu.org/bugs/index.php?46399>.
>
>@Mark: Can you check that there is nothing open from your POV?
>
>Workaround procedure: As long as there is not new version of m17n
>(i.e. >= 1.7.0) that can just be installed through package management,
>the fixes so far can be applied by creating a directory ~/.m17n.d and
>putting the file
><http://git.savannah.gnu.org/cgit/m17n/m17n-db.git/plain/FLT/HEBR-OTF.flt>
>there.

[-- Attachment #2: Aharoni bad nikud 2015-11-29 17.57.22.png --]
[-- Type: image/png, Size: 163490 bytes --]

bug#21846: 25.0.50; emacs hebrew script bidi nikud rendering bug

[Benjamin Riefenstahl]

Mark David writes:
> The nikud placement is good to excellent in the fonts tried so far.

Thank you for confirming that.

> However, I tried a few more, and found one that's terrible: Aharoni
>
> This font seems to systematically misplace not all but most base char
> + nikud combinations in my sample file.  (Screen shot attached.)  Any
> way to fix this?

When I install this from the package manager, I get a PostScript font,
not a TrueType font (you can verify that yourself with "fc-list 'Aharoni
CLM' file").  I don't know how PostScript fonts are handled by Emacs.
But, not surprisingly, it does not go through the FLT file that we have
been hacking on so far.

Note that AFAIK, PostScript fonts do not have the kind of metadata that
is needed for positioning individual glyphs as OpenType has, so all the
functionality comes from code.  Still, Harfbuzz can do it, so in
principle, Emacs can, too.  ;-)

bug#21846: 25.0.50; emacs hebrew script bidi nikud rendering bug

[Benjamin Riefenstahl]

Benjamin Riefenstahl writes:
> When I install this from the package manager, I get a PostScript font,
> not a TrueType font (you can verify that yourself with "fc-list
> 'Aharoni CLM' file").  I don't know how PostScript fonts are handled
> by Emacs.  But, not surprisingly, it does not go through the FLT file
> that we have been hacking on so far.

Interestingly, when I run this font and the test file through m17n-edit
(after some debugging why the font was not recognized :-(), it seems
fine there.  So in this case it does not seem to be a problem with m17n,
but actually with Emacs.

With m17n-edit, the text is run through COMBINING.flt (according to
MDEBUG_FLT=3), with Emacs, no FLT is triggered at all.  Which makes me
suspect that Emacs does not use m17n for this font at all (?).

bug#21846: 25.0.50; emacs hebrew script bidi nikud rendering bug

[Benjamin Riefenstahl]

Benjamin Riefenstahl writes:
> Interestingly, when I run this font and the test file through
> m17n-edit (after some debugging why the font was not recognized :-(),
> it seems fine there.  So in this case it does not seem to be a problem
> with m17n, but actually with Emacs.

For completeness the commands I used for testing:

  $ m17n-edit --font 'Aharoni CLM:weight=book' -s 200 emacs-test.txt
  $ emacs -Q --eval '(set-default-font "Aharoni CLM 20")' emacs-test.txt

bug#21846: 25.0.50; emacs hebrew script bidi nikud rendering bug

[Benjamin Riefenstahl]

Benjamin Riefenstahl writes:
> With m17n-edit, the text is run through COMBINING.flt (according to
> MDEBUG_FLT=3), with Emacs, no FLT is triggered at all.  Which makes me
> suspect that Emacs does not use m17n for this font at all (?).

So I armed myself with a debugger and tried to trace what happens.  My
command line is

  gdb --args ./emacs -Q --eval '(set-default-font "Aharoni CLM 20")' test.txt

The first wrong turn is in hebrew-shape-gstring
(lisp/language/hebrew.el).  This only calls the font-based shaping
function, if the font indicates that it supports OpenType (OTF) features
for Hebrew.  As a fallback, limited shaping is done in ELisp by looking
up precomposed characters.

When I stub this out for further testing, we get until ftfont_shape
(ftfont.c).  That function checks if there is any OTF support in the
font and only when it is there, it calls libm17n.  The next function in
the chain (ftfont_shape_by_flt) actually uses that OTF object that we
did not get in ftfont_shape to pass some font information.

There seem to be three possibilities:

* Don't fix, PostScript fonts are just not supported here.  Maybe instead "Aharoni CLM" could be converted to OpenType.
* Try to support non-OTF fonts in ftfont_shape_by_flt.  See how the example programs for libm17n do it.
* Reimplement what COMBINING.flt does in hebrew.el.

bug#21846: 25.0.50; emacs hebrew script bidi nikud rendering bug

[Mark H. David]

My vote

> On Dec 10, 2015, at 3:15 PM, Benjamin Riefenstahl <b.riefenstahl@turtle-trading.net> wrote:
> 
> * Try to support non-OTF fonts in ftfont_shape_by_flt.  See how the example programs for libm17n do it.

bug#21846: 25.0.50; emacs hebrew script bidi nikud rendering bug

[K. Handa]

In article <2EB9373F-2E83-46E6-84A6-FC0E28382BDA@yv.org>, "Mark H. David" <mhd@yv.org> writes:

> My vote
> > On Dec 10, 2015, at 3:15 PM, Benjamin Riefenstahl <b.riefenstahl@turtle-trading.net> wrote:
> > 
> > * Try to support non-OTF fonts in ftfont_shape_by_flt.  See how the example programs for libm17n do it.

I've just re-read the code.  It seems that supporting it without
breaking the backward compatiblity is not straitforward but possible.
I'll work on it soon.

---
K. Handa
handa@gnu.org

bug#21846: 25.0.50; emacs hebrew script bidi nikud rendering bug

[K. Handa]

I wrote:
> I've just re-read the code.  It seems that supporting it without
> breaking the backward compatiblity is not straitforward but possible.
> I'll work on it soon.

I committed changes to enable ftfont-base font-backend to handle wider
range of combining characters for non-OTF fonts.  So, hebrew
diacticicals should be rendered by "Aharoni CLM" font.

Could you verify it?

---
K. Handa
handa@gnu.org

bug#21846: 25.0.50; emacs hebrew script bidi nikud rendering bug

[Mark H. David]

Sorry for delayed response, just getting back to work.
Can you briefly explain what to update to test this change?
Thanks,
Mark

----- Original message -----
From: "K. Handa" <handa@gnu.org>
To: "K. Handa" <handa@gnu.org>
Cc: mhd@yv.org, b.riefenstahl@turtle-trading.net, 21846@debbugs.gnu.org, eliz@gnu.org
Subject: Re: 25.0.50; emacs hebrew script bidi nikud rendering bug
Date: Sun, 03 Jan 2016 21:03:50 +0900

I wrote:
> I've just re-read the code.  It seems that supporting it without
> breaking the backward compatiblity is not straitforward but possible.
> I'll work on it soon.

I committed changes to enable ftfont-base font-backend to handle wider
range of combining characters for non-OTF fonts.  So, hebrew
diacticicals should be rendered by "Aharoni CLM" font.

Could you verify it?

---
K. Handa
handa@gnu.org

bug#21846: 25.0.50; emacs hebrew script bidi nikud rendering bug

[Benjamin Riefenstahl]

"Mark H. David" writes:
> Sorry for delayed response, [...]

Ditto.

K. Handa) writes:
> I committed changes to enable ftfont-base font-backend to handle wider
> range of combining characters for non-OTF fonts.  So, hebrew
> diacticicals should be rendered by "Aharoni CLM" font.
>
> Could you verify it?

Looks good to me.  Mostly the same as with gedit, so we got parity there
;-)

"Mark H. David" writes:
> Can you briefly explain what to update to test this change?

The change is on the "master" branch.  It involves changes in the Lisp
code as well as on the C side.

Speaking of the branch, Handa, will you backport this to emacs-25, or
will we have to wait until 25.1 to get it in a regular release? ;-)

Anyway, from my side, it seems we can close this.

@Mark, do you see anything serious still missing?


benny

bug#21846: 25.0.50; emacs hebrew script bidi nikud rendering bug

[K. Handa]

In article <87mvsa7nyi.fsf@justinian.turtle-trading.net>, Benjamin Riefenstahl <b.riefenstahl@turtle-trading.net> writes:

> The change is on the "master" branch.  It involves changes in the Lisp
> code as well as on the C side.

> Speaking of the branch, Handa, will you backport this to emacs-25, or
> will we have to wait until 25.1 to get it in a regular release? ;-)

Displaying a script correctly is very important for users of that script.
So, I agree that it is better that the fix for bug#21846 is included in
the next release.

Eli, what is the current rule for back-porting a patch in trunk to a
branch for the next release?

FYI, my change had one bug, but have recently fixed.

---
K. Handa
handa@gnu.org

bug#21846: 25.0.50; emacs hebrew script bidi nikud rendering bug

[Eli Zaretskii]

> From: handa@gnu.org (K. Handa)
> Cc: mhd@yv.org,  21846@debbugs.gnu.org,  eliz@gnu.org
> Date: Fri, 22 Jan 2016 21:46:54 +0900
> 
> Eli, what is the current rule for back-porting a patch in trunk to a
> branch for the next release?
> 
> FYI, my change had one bug, but have recently fixed.

The emacs-25 branch is open for bugfixes, so I see no reason not to
backport from master.

Thanks.

bug#21846: 25.0.50; emacs hebrew script bidi nikud rendering bug

[Wolfgang Jenkner]

On Sun, Jan 03 2016, K. Handa wrote:

> I committed changes to enable ftfont-base font-backend to handle wider
> range of combining characters for non-OTF fonts.  So, hebrew
> diacticicals should be rendered by "Aharoni CLM" font.
>
> Could you verify it?

These changes (536f48e, 9835757, resp. the backports to emacs-25, viz.
4a3db0f, ad879b7) seem to break rendering of some Indic scripts.
Conjuncts are not composed anymore but rendered with virāma instead.
E.g., please see the Hindi entry of the HELLO file.

Apparently, the reason is this change

-      if (0x300 <= c1 && c1 <= 0x36F)
+      if (CHAR_HAS_CATEGORY (c1, '^'))

as, say, "DEVANAGARI SIGN VIRAMA" is excluded by the old condition but
included by the new one:

(category-set-mnemonics (char-category-set ?\u094d))
=> "^i"

bug#21846: 25.0.50; emacs hebrew script bidi nikud rendering bug

[handa]

In article <85pourn5g9.fsf@inode.at>, Wolfgang Jenkner <wjenkner@inode.at> writes:

> These changes (536f48e, 9835757, resp. the backports to emacs-25, viz.
> 4a3db0f, ad879b7) seem to break rendering of some Indic scripts.
> Conjuncts are not composed anymore but rendered with virāma instead.
> E.g., please see the Hindi entry of the HELLO file.

> Apparently, the reason is this change

> -      if (0x300 <= c1 && c1 <= 0x36F)
> +      if (CHAR_HAS_CATEGORY (c1, '^'))

> as, say, "DEVANAGARI SIGN VIRAMA" is excluded by the old condition but
> included by the new one:

Oops, right.  Have about this change?

---
K. Handa
handa@gnu.org

2016-03-21  handa  <handa@gnu.org>

	* ftfont.c (ftfont_shape_by_flt): For combining characters out
	of the range U+300...U+36F, use the "combining" FLT only with
	non-OTF fonts.

diff --git a/src/ftfont.c b/src/ftfont.c
index 505d508..17ccda4 100644
--- a/src/ftfont.c
+++ b/src/ftfont.c
@@ -2596,7 +2596,12 @@ ftfont_shape_by_flt (Lisp_Object lgstring, struct font *font,
 	 language information, and select a proper flt for them
 	 here.  */
       int c1 = LGLYPH_CHAR (LGSTRING_GLYPH (lgstring, 1));
-      if (CHAR_HAS_CATEGORY (c1, '^'))
+      /* For the combining characters in the range U+300..U+36F,
+	 "combining" is the sole FLT provided by the m17n-lib.  In
+	 addition, it is the sole FLT that can handle the other
+	 combining characters with non-OTF fonts.  */
+      if (0x300 <= c1 && c1 <= 0x36F
+	  || (! otf  && CHAR_HAS_CATEGORY (c1, '^')))
 	flt = mflt_get (msymbol ("combining"));
     }
   if (! flt && ! otf)

bug#21846: 25.0.50; emacs hebrew script bidi nikud rendering bug

[Wolfgang Jenkner]

On Mon, Mar 21 2016, handa wrote:

> Oops, right.  Have about this change?

Thank you!  I think that (together with this patch) the changes made in
response to this bug report have no effect on rendering with otf fonts?

In any case, I tested the patch and the issue with Devanāgarī is
certainly fixed.

bug#21846: 25.0.50; emacs hebrew script bidi nikud rendering bug

[Eli Zaretskii]

> From: Wolfgang Jenkner <wjenkner@inode.at>
> Date: Mon, 21 Mar 2016 20:05:55 +0100
> Cc: 21846@debbugs.gnu.org, mhd@yv.org, b.riefenstahl@turtle-trading.net
> 
> On Mon, Mar 21 2016, handa wrote:
> 
> > Oops, right.  Have about this change?
> 
> Thank you!  I think that (together with this patch) the changes made in
> response to this bug report have no effect on rendering with otf fonts?
> 
> In any case, I tested the patch and the issue with Devanāgarī is
> certainly fixed.

Thanks, I pushed Handa-san's patch to the emacs-25 branch.

bug#21846: 25.0.50; emacs hebrew script bidi nikud rendering bug

[Mark H. David]

Great work - thanks!

----- Original message -----
From: Eli Zaretskii <eliz@gnu.org>
To: Wolfgang Jenkner <wjenkner@inode.at>
Cc: handa@gnu.org, 21846-done@debbugs.gnu.org, mhd@yv.org, b.riefenstahl@turtle-trading.net
Subject: Re: bug#21846: 25.0.50; emacs hebrew script bidi nikud rendering bug
Date: Fri, 25 Mar 2016 12:42:13 +0300

> From: Wolfgang Jenkner <wjenkner@inode.at>
> Date: Mon, 21 Mar 2016 20:05:55 +0100
> Cc: 21846@debbugs.gnu.org, mhd@yv.org, b.riefenstahl@turtle-trading.net
> 
> On Mon, Mar 21 2016, handa wrote:
> 
> > Oops, right.  Have about this change?
> 
> Thank you!  I think that (together with this patch) the changes made in
> response to this bug report have no effect on rendering with otf fonts?
> 
> In any case, I tested the patch and the issue with Devanāgarī is
> certainly fixed.

Thanks, I pushed Handa-san's patch to the emacs-25 branch.

bug#21846: 25.0.50; emacs hebrew script bidi nikud rendering bug

[handa]

> > > Oops, right.  Have about this change?
> > 
> > Thank you!  I think that (together with this patch) the changes made in
> > response to this bug report have no effect on rendering with otf fonts?

Yes.

> > In any case, I tested the patch and the issue with Devanāgarī is
> > certainly fixed.

> Thanks, I pushed Handa-san's patch to the emacs-25 branch.

Thank you.

---
K. Handa
handa@gnu.org

bug#21846: 25.0.50; emacs hebrew script bidi nikud rendering bug

[K. Handa]

Hi,

> > This font seems to systematically misplace not all but most base char
> > + nikud combinations in my sample file.  (Screen shot attached.)  Any
> > way to fix this?

> When I install this from the package manager, I get a PostScript font,
> not a TrueType font (you can verify that yourself with "fc-list 'Aharoni
> CLM' file").  I don't know how PostScript fonts are handled by Emacs.
> But, not surprisingly, it does not go through the FLT file that we have
> been hacking on so far.

Could you tell me the download site of "Aharoni CLM" font?

---
K. Handa
handa@gnu.org

bug#21846: 25.0.50; emacs hebrew script bidi nikud rendering bug

[Mark David]

I originally brought up this font.  I did not ever download it (knowingly).
I thought it came built in with my Linux, with 'uname -a' output as follows:

Linux ubuntu 3.19.0-33-generic #38~14.04.1-Ubuntu SMP Fri Nov 6 18:17:28 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Maybe someone can figure out how it got into the distribution I'm using. I don't think I can, not easily.

-Mark




On 12/5/15, 4:55 AM, "K. Handa" <handa@gnu.org> wrote:

>Hi,
>
>> > This font seems to systematically misplace not all but most base char
>> > + nikud combinations in my sample file.  (Screen shot attached.)  Any
>> > way to fix this?
>
>> When I install this from the package manager, I get a PostScript font,
>> not a TrueType font (you can verify that yourself with "fc-list 'Aharoni
>> CLM' file").  I don't know how PostScript fonts are handled by Emacs.
>> But, not surprisingly, it does not go through the FLT file that we have
>> been hacking on so far.
>
>Could you tell me the download site of "Aharoni CLM" font?
>
>---
>K. Handa
>handa@gnu.org

bug#21846: 25.0.50; emacs hebrew script bidi nikud rendering bug

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I originally brought up this font.  I did not ever download it (knowingly).
  > I thought it came built in with my Linux, with 'uname -a' output as follows:

Linux is a kernel -- I don't think it contains fonts, except perhaps
one for the console.  Did you mean, "built in with my GNU/Linux
distro"?

See http://gnu.org/gnu/linux-and-gnu.html and
http://gnu.org/gnu/gnu-linux-faq.html, plus the history in
http://gnu.org/gnu/the-gnu-project.html.

-- 
Dr Richard Stallman
President, Free Software Foundation (gnu.org, fsf.org)
Internet Hall-of-Famer (internethalloffame.org)
Skype: No way! See stallman.org/skype.html.

bug#21846: 25.0.50; emacs hebrew script bidi nikud rendering bug

[K. Handa]

In article <E7BB124E-5727-44C9-AE42-7BA48A784884@yv.org>, Mark David <mhd@yv.org> writes:

> I originally brought up this font.  I did not ever download it (knowingly).
> I thought it came built in with my Linux, with 'uname -a' output as follows:

> Linux ubuntu 3.19.0-33-generic #38~14.04.1-Ubuntu SMP Fri Nov 6 18:17:28 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

> Maybe someone can figure out how it got into the distribution I'm using. I don't think I can, not easily.

Ubuntu has this page:
  https://help.ubuntu.com/community/HebrewLocalizationHowto
but it does not mention 'Aharoni CLM' font.

Do you have the packge libotf-bin installed?  If so, this command:
  % otfdump FONT-FILE-NAME > dump
dumps the OTF-related information of the specified font to the file
"dump".   Could you please send me that file?

---
K. Handa
handa@gnu.org

bug#21846: 25.0.50; emacs hebrew script bidi nikud rendering bug

[Benjamin Riefenstahl]

K. Handa writes:
> Could you tell me the download site of "Aharoni CLM" font?

  http://sourceforge.net/projects/culmus/files/culmus/0.130/

> Do you have the packge libotf-bin installed?  If so, this command:
>   % otfdump FONT-FILE-NAME > dump
> dumps the OTF-related information of the specified font to the file
> "dump".   Could you please send me that file?

  $ otfdump /usr/share/fonts/X11/Type1/AharoniCLM-Book.pfa
  otfdump: OTF-Error (File error): opening otf
  (/usr/share/fonts/X11/Type1/AharoniCLM-Book.pfa)