bug#34035: 26.1; Arabic shadda-kasrah renders incorrectly

bug#34035: 26.1; Arabic shadda-kasrah renders incorrectly

[Peter]

[-- Attachment #1: Type: text/plain, Size: 495 bytes --]

- Start emacs -Q
- C-\ arabic RET
- s ~ A (sin shadda kasrah) repeat a few times

The kasrah (short slash below the letter) should be shown below the
shadda (the squiggle above the letter) and above the letter, not below
it.

I've tried different fonts, this seems to be a problem with emacs
itself, other editors and firefox render the same text (copy-pasted)
correctly.

See attached screenshot for what it looks like in Emacs and in Firefox.

Thanks for any help with this!

Greetings, Peter

[-- Attachment #2: emacs-shadda-kasrah.png --]
[-- Type: image/png, Size: 73968 bytes --]

[-- Attachment #3: firefox-shadda-kasrah.png --]
[-- Type: image/png, Size: 3035 bytes --]

bug#34035: 26.1; Arabic shadda-kasrah renders incorrectly

[Eli Zaretskii]

> From: "Peter" <craven@gmx.net>
> Date: Thu, 10 Jan 2019 18:20:32 +0100
> 
> - Start emacs -Q
> - C-\ arabic RET
> - s ~ A (sin shadda kasrah) repeat a few times
> 
> The kasrah (short slash below the letter) should be shown below the
> shadda (the squiggle above the letter) and above the letter, not below
> it.
> 
> I've tried different fonts, this seems to be a problem with emacs
> itself, other editors and firefox render the same text (copy-pasted)
> correctly.
> 
> See attached screenshot for what it looks like in Emacs and in Firefox.

Thanks.  On my system, this is displayed correctly, with kasrah above
the letter.  So I don't think it's Emacs, I think it's the shaping
engine you are using.  As you didn't provide the information collected
by "M-x report-emacs-bug", I can only guess what is that shaping
engine: XFT and libflt, right?  Maybe you could try building the
harfbazz branch, which uses HarfBazz for shaping, I'd expect this
problem not to exist there.

bug#34035: 26.1; Arabic shadda-kasrah renders incorrectly

[Peter]

> Thanks.  On my system, this is displayed correctly, with kasrah above
> the letter.  So I don't think it's Emacs, I think it's the shaping
> engine you are using.  As you didn't provide the information collected
> by "M-x report-emacs-bug", I can only guess what is that shaping
> engine: XFT and libflt, right?  Maybe you could try building the
> harfbazz branch, which uses HarfBazz for shaping, I'd expect this
> problem not to exist there.

Thanks, I'll try this. The reason I didn't include that information is
that I sent the email from my "outer" emacs running exwm, but tried with
an "inner" emacs with emacs -Q (which is a rather inconvenient way of
running emacs ;).

Seems to be XFT and libflt:

  Configured using:
   'configure --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib
   --localstatedir=/var --with-x-toolkit=gtk3 --with-xft --with-modules
   'CFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong
   -fno-plt' CPPFLAGS=-D_FORTIFY_SOURCE=2
   LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now'
  
  Configured features:
  XPM JPEG TIFF GIF PNG RSVG IMAGEMAGICK SOUND GPM DBUS GSETTINGS NOTIFY
  ACL GNUTLS LIBXML2 FREETYPE M17N_FLT LIBOTF XFT ZLIB TOOLKIT_SCROLL_BARS
  GTK3 X11 MODULES THREADS LIBSYSTEMD LCMS2

I'll try that branch. Are there plans to move emacs head over to
harfbuzz at any point in the future?

Greetings and thanks for your help!

Peter

bug#34035: 26.1; Arabic shadda-kasrah renders incorrectly

[Eli Zaretskii]

> From: "Peter" <craven@gmx.net>
> Cc: 34035@debbugs.gnu.org
> Date: Thu, 10 Jan 2019 20:45:30 +0100
> 
> Are there plans to move emacs head over to harfbuzz at any point in
> the future?

Yes, we plan on merging the harfbuzz branch to master, eventually.

bug#34035: 26.1; Arabic shadda-kasrah renders incorrectly

[Peter]

> Yes, we plan on merging the harfbuzz branch to master, eventually.

Thanks, I've read up on the (very interesting) thread, I'll play around
with that branch tomorrow and send any problems I find that way.

Thank you again for your help in this!

Greetings, Peter

bug#34035: 26.1; Arabic shadda-kasrah renders incorrectly

[Stephen Berman]

On Thu, 10 Jan 2019 20:55:55 +0200 Eli Zaretskii <eliz@gnu.org> wrote:

>> From: "Peter" <craven@gmx.net>
>> Date: Thu, 10 Jan 2019 18:20:32 +0100
>> 
>> - Start emacs -Q
>> - C-\ arabic RET
>> - s ~ A (sin shadda kasrah) repeat a few times
>> 
>> The kasrah (short slash below the letter) should be shown below the
>> shadda (the squiggle above the letter) and above the letter, not below
>> it.
>> 
>> I've tried different fonts, this seems to be a problem with emacs
>> itself, other editors and firefox render the same text (copy-pasted)
>> correctly.
>> 
>> See attached screenshot for what it looks like in Emacs and in Firefox.
>
> Thanks.  On my system, this is displayed correctly, with kasrah above
> the letter.  So I don't think it's Emacs, I think it's the shaping
> engine you are using.  As you didn't provide the information collected
> by "M-x report-emacs-bug", I can only guess what is that shaping
> engine: XFT and libflt, right?  Maybe you could try building the
> harfbazz branch, which uses HarfBazz for shaping, I'd expect this
> problem not to exist there.

I believe the problem is not with the shaping engine but with the font:
I see the same problem on both builds from current master (with libotf)
and from the current harfbuzz branch using my default font, DejaVu Sans
Mono.  But when I switch the font to Symbola, the kasrah is correctly
displayed between the sin and the shadda, both on master and on
harfbuzz.  (Nevertheless, on both branches, after switching to Symbola,
describe-char surprisingly says this:

Composed with the following character(s) "ِّ" using this font:
  xft:-PfEd-DejaVu Sans-normal-normal-semicondensed-*-15-*-*-*-*-0-iso10646-1

Is this expected?)

Steve Berman

bug#34035: 26.1; Arabic shadda-kasrah renders incorrectly

[Eli Zaretskii]

> From: Stephen Berman <stephen.berman@gmx.net>
> Cc: "Peter" <craven@gmx.net>,  34035@debbugs.gnu.org
> Date: Fri, 11 Jan 2019 10:24:45 +0100
> 
> > Thanks.  On my system, this is displayed correctly, with kasrah above
> > the letter.  So I don't think it's Emacs, I think it's the shaping
> > engine you are using.  As you didn't provide the information collected
> > by "M-x report-emacs-bug", I can only guess what is that shaping
> > engine: XFT and libflt, right?  Maybe you could try building the
> > harfbazz branch, which uses HarfBazz for shaping, I'd expect this
> > problem not to exist there.
> 
> I believe the problem is not with the shaping engine but with the font:

The OP did say he tried different fonts, to no avail.  It would be
interesting to know which fonts were those.

> I see the same problem on both builds from current master (with libotf)
> and from the current harfbuzz branch using my default font, DejaVu Sans
> Mono.  But when I switch the font to Symbola, the kasrah is correctly
> displayed between the sin and the shadda, both on master and on
> harfbuzz.  (Nevertheless, on both branches, after switching to Symbola,
> describe-char surprisingly says this:
> 
> Composed with the following character(s) "ِّ" using this font:
>   xft:-PfEd-DejaVu Sans-normal-normal-semicondensed-*-15-*-*-*-*-0-iso10646-1
> 
> Is this expected?)

Please show the entire output of "C-x =" (I presume you invoke it on
the position of sin?).  FWIW, I don't see this here, but in my case
the default Courier New font is used for displaying this text.

bug#34035: 26.1; Arabic shadda-kasrah renders incorrectly

[Stephen Berman]

On Fri, 11 Jan 2019 11:30:56 +0200 Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Stephen Berman <stephen.berman@gmx.net>
>> Cc: "Peter" <craven@gmx.net>,  34035@debbugs.gnu.org
>> Date: Fri, 11 Jan 2019 10:24:45 +0100
>> 
>> > Thanks.  On my system, this is displayed correctly, with kasrah above
>> > the letter.  So I don't think it's Emacs, I think it's the shaping
>> > engine you are using.  As you didn't provide the information collected
>> > by "M-x report-emacs-bug", I can only guess what is that shaping
>> > engine: XFT and libflt, right?  Maybe you could try building the
>> > harfbazz branch, which uses HarfBazz for shaping, I'd expect this
>> > problem not to exist there.
>> 
>> I believe the problem is not with the shaping engine but with the font:
>
> The OP did say he tried different fonts, to no avail.  It would be
> interesting to know which fonts were those.
>
>> I see the same problem on both builds from current master (with libotf)
>> and from the current harfbuzz branch using my default font, DejaVu Sans
>> Mono.  But when I switch the font to Symbola, the kasrah is correctly
>> displayed between the sin and the shadda, both on master and on
>> harfbuzz.  (Nevertheless, on both branches, after switching to Symbola,
>> describe-char surprisingly says this:
>> 
>> Composed with the following character(s) "ِّ" using this font:
>>   xft:-PfEd-DejaVu Sans-normal-normal-semicondensed-*-15-*-*-*-*-0-iso10646-1
>> 
>> Is this expected?)
>
> Please show the entire output of "C-x =" 

I assume you meant `C-u C-x ='

>                                          (I presume you invoke it on
> the position of sin?).

Yes.  Here's the output on master:

==============================================================================
             position: 1 of 3 (0%), column: 0
            character: س‎ (displayed as س‎) (codepoint 1587, #o3063, #x633)
              charset: unicode (Unicode (ISO10646))
code point in charset: 0x0633
               script: arabic
               syntax: w 	which means: word
             category: .:Base, R:Right-to-left (strong), b:Arabic
             to input: type "s" with arabic input method
          buffer code: #xD8 #xB3
            file code: #xD8 #xB3 (encoded by coding system utf-8-unix)
              display: composed to form "سِّ" (see below)

Composed with the following character(s) "ِّ" using this font:
  xft:-PfEd-DejaVu Sans-normal-normal-semicondensed-*-15-*-*-*-*-0-iso10646-1
by these glyphs:
  [0 2 1587 1377 16 0 16 6 4 nil]
  [0 2 0 6022 0 -15 -10 13 -11 [-16 2 0]]

Character code properties: customize what to show
  name: ARABIC LETTER SEEN
  general-category: Lo (Letter, Other)
  decomposition: (1587) ('س')
==============================================================================

On the harfbuzz branch the output is the same except for the glyphs:

  [0 2 1587 6022 0 1 6 16 -8 [0 3 0]]
  [0 2 1587 1377 16 0 16 6 4 nil]

Note the second line here is identical to the first line of glyphs on
master.

Steve Berman

bug#34035: 26.1; Arabic shadda-kasrah renders incorrectly

[Eli Zaretskii]

> From: Stephen Berman <stephen.berman@gmx.net>
> Cc: craven@gmx.net,  34035@debbugs.gnu.org
> Date: Fri, 11 Jan 2019 10:47:11 +0100
> 
> > Please show the entire output of "C-x =" 
> 
> I assume you meant `C-u C-x ='
> 
> >                                          (I presume you invoke it on
> > the position of sin?).
> 
> Yes.  Here's the output on master:
> 
> ==============================================================================
>              position: 1 of 3 (0%), column: 0
>             character: س‎ (displayed as س‎) (codepoint 1587, #o3063, #x633)
>               charset: unicode (Unicode (ISO10646))
> code point in charset: 0x0633
>                script: arabic
>                syntax: w 	which means: word
>              category: .:Base, R:Right-to-left (strong), b:Arabic
>              to input: type "s" with arabic input method
>           buffer code: #xD8 #xB3
>             file code: #xD8 #xB3 (encoded by coding system utf-8-unix)
>               display: composed to form "سِّ" (see below)
> 
> Composed with the following character(s) "ِّ" using this font:
>   xft:-PfEd-DejaVu Sans-normal-normal-semicondensed-*-15-*-*-*-*-0-iso10646-1
> by these glyphs:
>   [0 2 1587 1377 16 0 16 6 4 nil]
>   [0 2 0 6022 0 -15 -10 13 -11 [-16 2 0]]
> 
> Character code properties: customize what to show
>   name: ARABIC LETTER SEEN
>   general-category: Lo (Letter, Other)
>   decomposition: (1587) ('س')
> ==============================================================================

This clearly says that Emacs uses DejaVu Sans for this grapheme
cluster, so I wonder what does "switch the font to Symbola" mean in
this case.  Can you tell what you did to switch to Symbola?

> On the harfbuzz branch the output is the same except for the glyphs:
> 
>   [0 2 1587 6022 0 1 6 16 -8 [0 3 0]]
>   [0 2 1587 1377 16 0 16 6 4 nil]
> 
> Note the second line here is identical to the first line of glyphs on
> master.

I would hardly call this "identical", since the offsets are also
different.  But I don't think that matters at this point.

bug#34035: 26.1; Arabic shadda-kasrah renders incorrectly

[Stephen Berman]

On Fri, 11 Jan 2019 12:34:12 +0200 Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Stephen Berman <stephen.berman@gmx.net>
>> Cc: craven@gmx.net,  34035@debbugs.gnu.org
>> Date: Fri, 11 Jan 2019 10:47:11 +0100
>> 
>> > Please show the entire output of "C-x =" 
>> 
>> I assume you meant `C-u C-x ='
>> 
>> >                                          (I presume you invoke it on
>> > the position of sin?).
>> 
>> Yes.  Here's the output on master:
>> 
>> ==============================================================================
>>              position: 1 of 3 (0%), column: 0
>>             character: س‎ (displayed as س‎) (codepoint 1587, #o3063, #x633)
>>               charset: unicode (Unicode (ISO10646))
>> code point in charset: 0x0633
>>                script: arabic
>>                syntax: w 	which means: word
>>              category: .:Base, R:Right-to-left (strong), b:Arabic
>>              to input: type "s" with arabic input method
>>           buffer code: #xD8 #xB3
>>             file code: #xD8 #xB3 (encoded by coding system utf-8-unix)
>>               display: composed to form "سِّ" (see below)
>> 
>> Composed with the following character(s) "ِّ" using this font:
>>   xft:-PfEd-DejaVu Sans-normal-normal-semicondensed-*-15-*-*-*-*-0-iso10646-1
>> by these glyphs:
>>   [0 2 1587 1377 16 0 16 6 4 nil]
>>   [0 2 0 6022 0 -15 -10 13 -11 [-16 2 0]]
>> 
>> Character code properties: customize what to show
>>   name: ARABIC LETTER SEEN
>>   general-category: Lo (Letter, Other)
>>   decomposition: (1587) ('س')
>> ==============================================================================
>
> This clearly says that Emacs uses DejaVu Sans for this grapheme
> cluster, so I wonder what does "switch the font to Symbola" mean in
> this case.  Can you tell what you did to switch to Symbola?

I started emacs with -Q, then I selected the "Set Default Font..." entry
from the Options menu, clicked on "Symbola Regular" and pressed the
select button.  After doing that, `C-u C-x =' on a character in
*scratch* shows this:

xft:-UFAS-Symbola-normal-normal-semicondensed-*-15-*-*-*-*-0-iso10646-1 (#x37)

I don't know why the Arabic character info is different, but the display
is correct with Symbola as default font, but not with DejaVu.

>> On the harfbuzz branch the output is the same except for the glyphs:
>> 
>>   [0 2 1587 6022 0 1 6 16 -8 [0 3 0]]
>>   [0 2 1587 1377 16 0 16 6 4 nil]
>> 
>> Note the second line here is identical to the first line of glyphs on
>> master.
>
> I would hardly call this "identical", since the offsets are also
> different.  But I don't think that matters at this point.

I don't know if it's significant, but it's only first glyph line from
the harfbuzz build:
[0 2 1587 6022 0 1 6 16 -8 [0 3 0]]
that differs from the second glyph line from the master build:
[0 2 0 6022 0 -15 -10 13 -11 [-16 2 0]]
The second glyph line from the harfbuzz build:
[0 2 1587 1377 16 0 16 6 4 nil]
is identical to the first glyph line from the master build:
[0 2 1587 1377 16 0 16 6 4 nil]

Steve Berman

bug#34035: 26.1; Arabic shadda-kasrah renders incorrectly

[Eli Zaretskii]

> From: Stephen Berman <stephen.berman@gmx.net>
> Cc: craven@gmx.net,  34035@debbugs.gnu.org
> Date: Fri, 11 Jan 2019 11:54:06 +0100
> 
> > This clearly says that Emacs uses DejaVu Sans for this grapheme
> > cluster, so I wonder what does "switch the font to Symbola" mean in
> > this case.  Can you tell what you did to switch to Symbola?
> 
> I started emacs with -Q, then I selected the "Set Default Font..." entry
> from the Options menu, clicked on "Symbola Regular" and pressed the
> select button.  After doing that, `C-u C-x =' on a character in
> *scratch* shows this:
> 
> xft:-UFAS-Symbola-normal-normal-semicondensed-*-15-*-*-*-*-0-iso10646-1 (#x37)

Symbola supports only 10 characters from the Arabic block, so Emacs
selects a different font for these characters.  What I don't
understand is why the default font has any effect on how characters in
a non-default font are displayed.  CC'ing Handa-san, in the hope that
he might have some insights.

> >>   [0 2 1587 6022 0 1 6 16 -8 [0 3 0]]
> >>   [0 2 1587 1377 16 0 16 6 4 nil]
> >> 
> >> Note the second line here is identical to the first line of glyphs on
> >> master.
> >
> > I would hardly call this "identical", since the offsets are also
> > different.  But I don't think that matters at this point.
> 
> I don't know if it's significant, but it's only first glyph line from
> the harfbuzz build:
> [0 2 1587 6022 0 1 6 16 -8 [0 3 0]]
> that differs from the second glyph line from the master build:
> [0 2 0 6022 0 -15 -10 13 -11 [-16 2 0]]
> The second glyph line from the harfbuzz build:
> [0 2 1587 1377 16 0 16 6 4 nil]
> is identical to the first glyph line from the master build:
> [0 2 1587 1377 16 0 16 6 4 nil]

The identical line describes the base character (1587 is its Unicode
codepoint) in the same font, so it's small wonder that it is the same
with both shaping engines.  The different lines describe the
shadda-kasra diacritics.  (I'm mildly bothered by the fact that the
harfbuzz branch displays the information in reverse order, it might
mean some bug there.)

You can read more about the values displayed there in the doc string
of composition-get-gstring.

bug#34035: 26.1; Arabic shadda-kasrah renders incorrectly

[Stephen Berman]

On Fri, 11 Jan 2019 15:30:55 +0200 Eli Zaretskii <eliz@gnu.org> wrote:

>> From: Stephen Berman <stephen.berman@gmx.net>
>> Cc: craven@gmx.net,  34035@debbugs.gnu.org
>> Date: Fri, 11 Jan 2019 11:54:06 +0100
>> 
>> > This clearly says that Emacs uses DejaVu Sans for this grapheme
>> > cluster, so I wonder what does "switch the font to Symbola" mean in
>> > this case.  Can you tell what you did to switch to Symbola?
>> 
>> I started emacs with -Q, then I selected the "Set Default Font..." entry
>> from the Options menu, clicked on "Symbola Regular" and pressed the
>> select button.  After doing that, `C-u C-x =' on a character in
>> *scratch* shows this:
>> 
>> xft:-UFAS-Symbola-normal-normal-semicondensed-*-15-*-*-*-*-0-iso10646-1 (#x37)
>
> Symbola supports only 10 characters from the Arabic block, so Emacs
> selects a different font for these characters.  

That's what I suspected.  And I've now found out what the crucial
difference seems to be: I went through the fonts installed on my system,
and there are only two families that display Arabic characters: DejaVu
and Amiri, and only DejaVu Sans displays the shadda-kasrah combination
correctly; in particular my default font, DejaVu Sans Mono, displays it
incorrectly.  Surprisingly, Amiri, which is specifically designed for
Arabic, also displays it incorrectly, with the kasrah below the sin.
With all the other fonts, Arabic characters are displayed using either
Amiri or DejaVu, but of the latter, only a few (including Symbola) use
DejaVu Sans, most (even non-monospaced fonts) use DejaVu Sans Mono. 

>> I don't know if it's significant, but it's only first glyph line from
>> the harfbuzz build:
>> [0 2 1587 6022 0 1 6 16 -8 [0 3 0]]
>> that differs from the second glyph line from the master build:
>> [0 2 0 6022 0 -15 -10 13 -11 [-16 2 0]]
>> The second glyph line from the harfbuzz build:
>> [0 2 1587 1377 16 0 16 6 4 nil]
>> is identical to the first glyph line from the master build:
>> [0 2 1587 1377 16 0 16 6 4 nil]
>
> The identical line describes the base character (1587 is its Unicode
> codepoint) in the same font, so it's small wonder that it is the same
> with both shaping engines.  The different lines describe the
> shadda-kasra diacritics.  

That's also what I thought.

>                           (I'm mildly bothered by the fact that the
> harfbuzz branch displays the information in reverse order, it might
> mean some bug there.)

I wondered about that too.

> You can read more about the values displayed there in the doc string
> of composition-get-gstring.

Thanks for the pointer.

Steve Berman

bug#34035: 26.1; Arabic shadda-kasrah renders incorrectly

[Stefan Kangas]

[Resending using "Reply to all".]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: "Peter" <craven@gmx.net>
>> Date: Thu, 10 Jan 2019 18:20:32 +0100
>>
>> - Start emacs -Q
>> - C-\ arabic RET
>> - s ~ A (sin shadda kasrah) repeat a few times
>>
>> The kasrah (short slash below the letter) should be shown below the
>> shadda (the squiggle above the letter) and above the letter, not below
>> it.

I've tried this on the master branch.  I think I'm seeing that the
kasrah is shown below the shadda but above the Arabic letter s.  But I
could be mistaken since my knowledge of the Arabic script is just barely
better than non-existent.

>> I've tried different fonts, this seems to be a problem with emacs
>> itself, other editors and firefox render the same text (copy-pasted)
>> correctly.
>>
>> See attached screenshot for what it looks like in Emacs and in Firefox.
>
> Thanks.  On my system, this is displayed correctly, with kasrah above
> the letter.  So I don't think it's Emacs, I think it's the shaping
> engine you are using.  As you didn't provide the information collected
> by "M-x report-emacs-bug", I can only guess what is that shaping
> engine: XFT and libflt, right?  Maybe you could try building the
> harfbazz branch, which uses HarfBazz for shaping, I'd expect this
> problem not to exist there.

So now Emacs 27.1 is released with harfbuzz support.  Could you please
try this again using that and see if you can still reproduce this issue
there?

If I don't hear back from you within a couple of weeks, I'll just
assume that this has been fixed and close this bug.

Best regards,
Stefan Kangas

bug#34035: 26.1; Arabic shadda-kasrah renders incorrectly

[Basil L. Contovounesios]

[-- Attachment #1: Type: text/plain, Size: 364 bytes --]

Stefan Kangas <stefan@marxist.se> writes:

> So now Emacs 27.1 is released with harfbuzz support.  Could you please
> try this again using that and see if you can still reproduce this issue
> there?

I can still reproduce all of the above on both emacs-27 and master:

0. emacs -Q
1. C-x C-= C-= C-=
2. C-\ arabic RET
3. s ~ A

The kasrah is shown below the sin:


[-- Attachment #2: 0.png --]
[-- Type: image/png, Size: 2165 bytes --]

[-- Attachment #3: Type: text/plain, Size: 1328 bytes --]


4. C-b
5. C-u C-x =

--8<---------------cut here---------------start------------->8---
             position: 146 of 148 (98%), column: 0
            character: س‎ (displayed as س‎) (codepoint 1587, #o3063, #x633)
              charset: unicode (Unicode (ISO10646))
code point in charset: 0x0633
               script: arabic
               syntax: w 	which means: word
             category: .:Base, R:Right-to-left (strong), b:Arabic
             to input: type "s" with arabic input method
          buffer code: #xD8 #xB3
            file code: #xD8 #xB3 (encoded by coding system utf-8-unix)
              display: composed to form "سِّ" (see below)

Composed with the following character(s) "ِّ" using this font:
  xfthb:-PfEd-DejaVu Sans Mono-normal-normal-normal-*-26-*-*-*-m-0-iso10646-1
by these glyphs:
  [0 2 1616 1153 16 4 12 0 4 [0 6 0]]
  [0 2 1617 1154 16 3 12 23 -15 [0 2 0]]
  [0 2 1587 1129 16 -4 15 10 7 nil]

Character code properties: customize what to show
  name: ARABIC LETTER SEEN
  general-category: Lo (Letter, Other)
  decomposition: (1587) ('س')

There are text properties here:
  fontified            nil
--8<---------------cut here---------------end--------------->8---

6. M-x set-frame-font RET symbola RET

The kasrah is shown above the sin:


[-- Attachment #4: 1.png --]
[-- Type: image/png, Size: 2220 bytes --]

[-- Attachment #5: Type: text/plain, Size: 3027 bytes --]


7. C-u C-x =

--8<---------------cut here---------------start------------->8---
             position: 146 of 148 (98%), column: 0
            character: س‎ (displayed as س‎) (codepoint 1587, #o3063, #x633)
              charset: unicode (Unicode (ISO10646))
code point in charset: 0x0633
               script: arabic
               syntax: w 	which means: word
             category: .:Base, R:Right-to-left (strong), b:Arabic
             to input: type "s" with arabic input method
          buffer code: #xD8 #xB3
            file code: #xD8 #xB3 (encoded by coding system utf-8-unix)
              display: composed to form "سِّ" (see below)

Composed with the following character(s) "ِّ" using this font:
  xfthb:-PfEd-DejaVu Sans-normal-normal-semicondensed-*-26-*-*-*-*-0-iso10646-1
by these glyphs:
  [0 2 1616 6022 0 2 10 27 -15 [1 5 0]]
  [0 2 1617 1377 29 1 27 10 7 nil]

Character code properties: customize what to show
  name: ARABIC LETTER SEEN
  general-category: Lo (Letter, Other)
  decomposition: (1587) ('س')

There are text properties here:
  fontified            t
--8<---------------cut here---------------end--------------->8---

My Emacs version details follow.

HTH,

-- 
Basil

In GNU Emacs 27.1.50 (build 1, x86_64-pc-linux-gnu, X toolkit, Xaw3d scroll bars)
 of 2020-08-13 built on tabos
Repository revision: 69568674b3bb5b5f65f2c1342aded205a00f696a
Repository branch: emacs-27
Windowing system distributor 'The X.Org Foundation', version 11.0.12008000
System Description: Debian GNU/Linux bullseye/sid

Configured using:
 'configure 'CC=ccache gcc' 'CFLAGS=-O0 -g3 -ggdb -gdwarf-4'
 --config-cache --prefix=/home/blc/.local --program-suffix=27
 --enable-checking=yes,glyphs --enable-check-lisp-object-type
 --with-x-toolkit=lucid --with-file-notification=yes --with-x'

Configured features:
XAW3D XPM JPEG TIFF GIF PNG RSVG SOUND GPM DBUS GSETTINGS GLIB NOTIFY
INOTIFY ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE HARFBUZZ M17N_FLT LIBOTF
XFT ZLIB TOOLKIT_SCROLL_BARS LUCID X11 XDBE XIM MODULES THREADS
LIBSYSTEMD JSON PDUMPER LCMS2 GMP

Important settings:
  value of $LANG: en_IE.UTF-8
  locale-coding-system: utf-8-unix

In GNU Emacs 28.0.50 (build 1, x86_64-pc-linux-gnu, X toolkit, cairo version 1.16.0, Xaw3d scroll bars)
 of 2020-08-18 built on tabos
Repository revision: 92ec47981acd353658782ce2e094e9f270d8bd8d
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12008000
System Description: Debian GNU/Linux bullseye/sid

Configured using:
 'configure 'CC=ccache gcc' 'CFLAGS=-O2 -march=native' --config-cache
 --prefix=/home/blc/.local --with-x-toolkit=lucid
 --with-file-notification=yes --with-x'

Configured features:
XAW3D XPM JPEG TIFF GIF PNG RSVG CAIRO SOUND GPM DBUS GSETTINGS GLIB
NOTIFY INOTIFY ACL LIBSELINUX GNUTLS LIBXML2 FREETYPE HARFBUZZ M17N_FLT
LIBOTF ZLIB TOOLKIT_SCROLL_BARS LUCID X11 XDBE XIM MODULES THREADS
LIBSYSTEMD JSON PDUMPER LCMS2

bug#34035: 26.1; Arabic shadda-kasrah renders incorrectly

[Stephen Berman]

On Wed, 19 Aug 2020 09:01:06 +0100 "Basil L. Contovounesios" <contovob@tcd.ie> wrote:

> Stefan Kangas <stefan@marxist.se> writes:
>
>> So now Emacs 27.1 is released with harfbuzz support.  Could you please
>> try this again using that and see if you can still reproduce this issue
>> there?
>
> I can still reproduce all of the above on both emacs-27 and master:

This seems to be a font issue independent of Emacs, e.g., I also see the
same thing in LibreOffice Writer: with DejaVu Sans the kasrah is
correctly displayed between the shadda and the seen, and with DejaVu
Sans Mono, the kasrah is incorrectly displayed below the seen.

Steve Berman

bug#34035: 26.1; Arabic shadda-kasrah renders incorrectly

[Stefan Kangas]

Stephen Berman <stephen.berman@gmx.net> writes:

> On Wed, 19 Aug 2020 09:01:06 +0100 "Basil L. Contovounesios" <contovob@tcd.ie> wrote:
>
>> Stefan Kangas <stefan@marxist.se> writes:
>>
>>> So now Emacs 27.1 is released with harfbuzz support.  Could you please
>>> try this again using that and see if you can still reproduce this issue
>>> there?
>>
>> I can still reproduce all of the above on both emacs-27 and master:
>
> This seems to be a font issue independent of Emacs, e.g., I also see the
> same thing in LibreOffice Writer: with DejaVu Sans the kasrah is
> correctly displayed between the shadda and the seen, and with DejaVu
> Sans Mono, the kasrah is incorrectly displayed below the seen.

I can confirm that the font Emacs uses to correctly display this for me
in "emacs -Q" is:

ftcrhb:-PfEd-DejaVu Sans-normal-normal-normal-*-48-*-*-*-*-0-iso10646-1

I also tried LibreOffice Writer, fonts "FreeSans" and "DejaVu Sans Mono"
seem to be broken, while "DejaVu Sans" displays it correctly.

Does that mean that this is a bug/missing feature in the fonts, and not
in Emacs?

Best regards,
Stefan Kangas

bug#34035: 26.1; Arabic shadda-kasrah renders incorrectly

[Basil L. Contovounesios]

Stefan Kangas <stefan@marxist.se> writes:

> Stephen Berman <stephen.berman@gmx.net> writes:
>
>> This seems to be a font issue independent of Emacs, e.g., I also see the
>> same thing in LibreOffice Writer: with DejaVu Sans the kasrah is
>> correctly displayed between the shadda and the seen, and with DejaVu
>> Sans Mono, the kasrah is incorrectly displayed below the seen.
>
> I can confirm that the font Emacs uses to correctly display this for me
> in "emacs -Q" is:
>
> ftcrhb:-PfEd-DejaVu Sans-normal-normal-normal-*-48-*-*-*-*-0-iso10646-1
>
> I also tried LibreOffice Writer, fonts "FreeSans" and "DejaVu Sans Mono"
> seem to be broken, while "DejaVu Sans" displays it correctly.
>
> Does that mean that this is a bug/missing feature in the fonts, and not
> in Emacs?

Sounds like it, though I'm curious whether Eli's concerns in
https://debbugs.gnu.org/34035#35 still apply.

Thanks,

-- 
Basil

bug#34035: 26.1; Arabic shadda-kasrah renders incorrectly

[Eli Zaretskii]

> From: "Basil L. Contovounesios" <contovob@tcd.ie>
> Date: Wed, 19 Aug 2020 11:48:20 +0100
> Cc: 34035@debbugs.gnu.org, Stephen Berman <stephen.berman@gmx.net>,
>  Peter <craven@gmx.net>
> 
> > Does that mean that this is a bug/missing feature in the fonts, and not
> > in Emacs?
> 
> Sounds like it, though I'm curious whether Eli's concerns in
> https://debbugs.gnu.org/34035#35 still apply.

You mean, the order of showing the glyphs in the grapheme cluster?  I
eventually convinced myself that this is a side effect of how HarfBuzz
works with R2L compositions, not a bug.

bug#34035: 26.1; Arabic shadda-kasrah renders incorrectly

[Basil L. Contovounesios]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: "Basil L. Contovounesios" <contovob@tcd.ie>
>> Date: Wed, 19 Aug 2020 11:48:20 +0100
>> Cc: 34035@debbugs.gnu.org, Stephen Berman <stephen.berman@gmx.net>,
>>  Peter <craven@gmx.net>
>> 
>> > Does that mean that this is a bug/missing feature in the fonts, and not
>> > in Emacs?
>> 
>> Sounds like it, though I'm curious whether Eli's concerns in
>> https://debbugs.gnu.org/34035#35 still apply.
>
> You mean, the order of showing the glyphs in the grapheme cluster?  I
> eventually convinced myself that this is a side effect of how HarfBuzz
> works with R2L compositions, not a bug.

That and "why the default font has any effect on how characters in a
non-default font are displayed".

-- 
Basil

bug#34035: 26.1; Arabic shadda-kasrah renders incorrectly

[Eli Zaretskii]

> From: "Basil L. Contovounesios" <contovob@tcd.ie>
> Cc: stefan@marxist.se,  34035@debbugs.gnu.org,  stephen.berman@gmx.net,
>   craven@gmx.net
> Date: Wed, 19 Aug 2020 17:20:57 +0100
> 
> > You mean, the order of showing the glyphs in the grapheme cluster?  I
> > eventually convinced myself that this is a side effect of how HarfBuzz
> > works with R2L compositions, not a bug.
> 
> That and "why the default font has any effect on how characters in a
> non-default font are displayed".

Ah, that was me not paying attention: using Symbola as the default
font changed the font used for Arabic to DejaVu Sans, instead of
DejaVu Sans Mono.

bug#34035: 26.1; Arabic shadda-kasrah renders incorrectly

[Stefan Kangas]

"Basil L. Contovounesios" <contovob@tcd.ie> writes:

> Stefan Kangas <stefan@marxist.se> writes:
>
>> Stephen Berman <stephen.berman@gmx.net> writes:
>>
>>> This seems to be a font issue independent of Emacs, e.g., I also see the
>>> same thing in LibreOffice Writer: with DejaVu Sans the kasrah is
>>> correctly displayed between the shadda and the seen, and with DejaVu
>>> Sans Mono, the kasrah is incorrectly displayed below the seen.
>>
>> I can confirm that the font Emacs uses to correctly display this for me
>> in "emacs -Q" is:
>>
>> ftcrhb:-PfEd-DejaVu Sans-normal-normal-normal-*-48-*-*-*-*-0-iso10646-1
>>
>> I also tried LibreOffice Writer, fonts "FreeSans" and "DejaVu Sans Mono"
>> seem to be broken, while "DejaVu Sans" displays it correctly.
>>
>> Does that mean that this is a bug/missing feature in the fonts, and not
>> in Emacs?
>
> Sounds like it, though I'm curious whether Eli's concerns in
> https://debbugs.gnu.org/34035#35 still apply.

The discussion here concluded that there is indeed no (remaining) bug
here.  I'm therefore closing this bug report.

bug#34035: 26.1; Arabic shadda-kasrah renders incorrectly

[Eli Zaretskii]

> From: "Basil L. Contovounesios" <contovob@tcd.ie>
> Cc: Eli Zaretskii <eliz@gnu.org>,  34035@debbugs.gnu.org,  Peter
>  <craven@gmx.net>
> Date: Wed, 19 Aug 2020 09:01:06 +0100
> 
> The kasrah is shown below the sin:
> 
> [...]
> 
> Composed with the following character(s) "ِّ" using this font:
>   xfthb:-PfEd-DejaVu Sans Mono-normal-normal-normal-*-26-*-*-*-m-0-iso10646-1
> by these glyphs:
>   [0 2 1616 1153 16 4 12 0 4 [0 6 0]]
>   [0 2 1617 1154 16 3 12 23 -15 [0 2 0]]
>   [0 2 1587 1129 16 -4 15 10 7 nil]
> 
> Character code properties: customize what to show
>   name: ARABIC LETTER SEEN
>   general-category: Lo (Letter, Other)
>   decomposition: (1587) ('س')
> 
> [...]
> 
> The kasrah is shown above the sin:
> 
> [...]
> 
> Composed with the following character(s) "ِّ" using this font:
>   xfthb:-PfEd-DejaVu Sans-normal-normal-semicondensed-*-26-*-*-*-*-0-iso10646-1
> by these glyphs:
>   [0 2 1616 6022 0 2 10 27 -15 [1 5 0]]
>   [0 2 1617 1377 29 1 27 10 7 nil]
> 
> Character code properties: customize what to show
>   name: ARABIC LETTER SEEN
>   general-category: Lo (Letter, Other)
>   decomposition: (1587) ('س')

Which seems to clearly indicate that this _is_ font-dependent, right?

Moreover, it seems also to hint on the reason for the issue: the
correct display uses only 2 glyphs, whereas the incorrect display uses
3 glyphs.  Which means -- and that matches my observations on my
systems -- that the "good" font has a precomposed glyph for
shadda-kasrah, while the "bad" font doesn't.  And the composition data
in the latter case indicates that we were told to display the kasrah
below the base character (the descent value is positive).

Can someone please see what HarfBuzz's hb-view produces from these
glyphs, with the same fonts as you see in Emacs?  If hb-view produces
the same display for the same fonts, then it's not an Emacs problem,
and we should ask the HarfBuzz developers what, if anything, HarfBuzz
can do better for the problematic fonts.  And if hb-view does better
than Emacs, then we should ask the HarfBuzz developers to help us
understand what we do incorrectly in this case.

bug#34035: 26.1; Arabic shadda-kasrah renders incorrectly

[Basil L. Contovounesios]

[-- Attachment #1: Type: text/plain, Size: 1118 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

> Which seems to clearly indicate that this _is_ font-dependent, right?
>
> Moreover, it seems also to hint on the reason for the issue: the
> correct display uses only 2 glyphs, whereas the incorrect display uses
> 3 glyphs.  Which means -- and that matches my observations on my
> systems -- that the "good" font has a precomposed glyph for
> shadda-kasrah, while the "bad" font doesn't.  And the composition data
> in the latter case indicates that we were told to display the kasrah
> below the base character (the descent value is positive).

Makes sense.

> Can someone please see what HarfBuzz's hb-view produces from these
> glyphs, with the same fonts as you see in Emacs?  If hb-view produces
> the same display for the same fonts, then it's not an Emacs problem,
> and we should ask the HarfBuzz developers what, if anything, HarfBuzz
> can do better for the problematic fonts.  And if hb-view does better
> than Emacs, then we should ask the HarfBuzz developers to help us
> understand what we do incorrectly in this case.

Emacs and hb-view seem to be in agreement:


[-- Attachment #2: hb-view.png --]
[-- Type: image/png, Size: 25313 bytes --]

[-- Attachment #3: Type: text/plain, Size: 20 bytes --]


Thanks,

-- 
Basil

bug#34035: 26.1; Arabic shadda-kasrah renders incorrectly

[Eli Zaretskii]

> From: "Basil L. Contovounesios" <contovob@tcd.ie>
> Cc: stefan@marxist.se,  34035@debbugs.gnu.org,  craven@gmx.net
> Date: Wed, 19 Aug 2020 17:18:36 +0100
> 
> > Can someone please see what HarfBuzz's hb-view produces from these
> > glyphs, with the same fonts as you see in Emacs?  If hb-view produces
> > the same display for the same fonts, then it's not an Emacs problem,
> > and we should ask the HarfBuzz developers what, if anything, HarfBuzz
> > can do better for the problematic fonts.  And if hb-view does better
> > than Emacs, then we should ask the HarfBuzz developers to help us
> > understand what we do incorrectly in this case.
> 
> Emacs and hb-view seem to be in agreement:

Thanks.  I posted a question to the HarfBuzz list; you can follow it
at

  https://lists.freedesktop.org/archives/harfbuzz/2020-August/007540.html

bug#34035: 26.1; Arabic shadda-kasrah renders incorrectly

[Basil L. Contovounesios]

tags 34035 notabug
close 34035
quit

Eli Zaretskii <eliz@gnu.org> writes:

>> From: "Basil L. Contovounesios" <contovob@tcd.ie>
>> Cc: stefan@marxist.se,  34035@debbugs.gnu.org,  craven@gmx.net
>> Date: Wed, 19 Aug 2020 17:18:36 +0100
>> 
>> Emacs and hb-view seem to be in agreement:
>
> Thanks.  I posted a question to the HarfBuzz list; you can follow it
> at
>
>   https://lists.freedesktop.org/archives/harfbuzz/2020-August/007540.html

Thanks.  So far the consensus there seems to be that this is intended
and not wrong:

Richard Wordingham <richard.wordingham@ntlworld.com> writes:

> There is no evidence of a fault.  The TUS claims that
> kesra between shadda and the consonant came in with metal type.
> Basically, there are two different styles, one with kesra below the
> consonant and one with it below the shadda.  I don't have details of
> what is preferred where, or its social stratification.
>
> Wright (available at www.ghazali.org/arabic/WrightArabicGrammarVol1.pdf
> ), in §11 Remark (e) ɡives a slightly different story and some attempt
> at an indication of regional variation.

Jonathan Kew <jfkthame@gmail.com> writes:

> Yes; how the combination renders is up to the font.
>
> Some fonts even provide a user-selectable feature to control it, e.g. 
> https://software.sil.org/scheherazade/support/smart-font-features/.

I'm therefore closing this report.

-- 
Basil

bug#34035: 26.1; Arabic shadda-kasrah renders incorrectly

[James Cloos]

i bet this is the same as the regression i posted a while back where
cr+hb fails to handle fully mono fonts whereas the xft-based code does
handle them.

i suspect m17n-lib had the necessary code, whereas hb lazily only cares
aboutfonts with zero-width accents or with routines *in* each font
correctly to combine glyphs.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 0x997A9F17ED7DAEA6

bug#34035: 26.1; Arabic shadda-kasrah renders incorrectly

[Eli Zaretskii]

> From: James Cloos <cloos@jhcloos.com>
> Cc: "Basil L. Contovounesios" <contovob@tcd.ie>,  34035@debbugs.gnu.org,
>   stefan@marxist.se,  craven@gmx.net
> Date: Sun, 23 Aug 2020 02:41:09 -0400
> 
> i bet this is the same as the regression i posted a while back where
> cr+hb fails to handle fully mono fonts whereas the xft-based code does
> handle them.
> 
> i suspect m17n-lib had the necessary code, whereas hb lazily only cares
> aboutfonts with zero-width accents or with routines *in* each font
> correctly to combine glyphs.

I don't understand why you consider this a regression.  For the
shadda-kasrah case, the experts seem to unanimously say that both
renderings are possible and correct, and which one is used is a
stylistic preference.

bug#34035: 26.1; Arabic shadda-kasrah renders incorrectly

[Stephen Berman]

On Sun, 23 Aug 2020 10:15:19 +0300 Eli Zaretskii <eliz@gnu.org> wrote:

>> From: James Cloos <cloos@jhcloos.com>
>> Cc: "Basil L. Contovounesios" <contovob@tcd.ie>,  34035@debbugs.gnu.org,
>>   stefan@marxist.se,  craven@gmx.net
>> Date: Sun, 23 Aug 2020 02:41:09 -0400
>>
>> i bet this is the same as the regression i posted a while back where
>> cr+hb fails to handle fully mono fonts whereas the xft-based code does
>> handle them.
>>
>> i suspect m17n-lib had the necessary code, whereas hb lazily only cares
>> aboutfonts with zero-width accents or with routines *in* each font
>> correctly to combine glyphs.
>
> I don't understand why you consider this a regression.  For the
> shadda-kasrah case, the experts seem to unanimously say that both
> renderings are possible and correct, and which one is used is a
> stylistic preference.

I agree with the latter sentence (and indeed this thread taught me that
both renderings are legitimate), yet it seems unlikely to me that
e.g. the designers of DejaVu Sans and DejaVu Sans Mono deliberately made
this stylistic distinction between these two fonts.  And if not, and if
these two fonts display shadda-kasra combinations the same under
m17n-lib (I cannot test that right now), then that would indeed indicate
a bug in HarfBuzz/Cairo.

Steve Berman

bug#34035: 26.1; Arabic shadda-kasrah renders incorrectly

[Eli Zaretskii]

> From: Stephen Berman <stephen.berman@gmx.net>
> Cc: James Cloos <cloos@jhcloos.com>,  contovob@tcd.ie,
>   34035@debbugs.gnu.org,  stefan@marxist.se,  craven@gmx.net
> Date: Sun, 23 Aug 2020 11:26:53 +0200
> 
> I agree with the latter sentence (and indeed this thread taught me that
> both renderings are legitimate), yet it seems unlikely to me that
> e.g. the designers of DejaVu Sans and DejaVu Sans Mono deliberately made
> this stylistic distinction between these two fonts.  And if not, and if
> these two fonts display shadda-kasra combinations the same under
> m17n-lib (I cannot test that right now), then that would indeed indicate
> a bug in HarfBuzz/Cairo.

I very much doubt that there's such a bug in HarfBuzz, but if this is
indeed a HarfBuzz bug, it should be reported to the HarfBuzz
developers.  So far none of them spoke up in the thread I started on
their list (which is one more evidence that there's no bug here).