bug#10895: Quirky behaviours with Arabic text

bug#10895: Quirky behaviours with Arabic text

[sergei karhof]

I want to report some quirky behaviours of Emacs (devel 24.0.93 version,
running under Windows 7), when dealing with Arabic text:

1) in some fonts (e.g. courier New), the vowel marks (harakaat) are
not displayed, despite the fact that the texts contains them. However,
if the cursor passes over the relevant text, moving in the
right-to-left direction, the vowel marks become visible. Curiously, if
the cursor passes back on the text, moving in the left-to-right
direction, the vowel marks disappear.
This problem manifests itself in different ways according to the font used.
With other fonts, the problem is only partial: only part of the
diacritic signs is visible, and by passing over them with the cursor
(from right to left), they become fully visible. When passing over the
text in the opposite direction, they again become half-visible.

2) the rendering of the vowel marks is poor even when they are
displayed. For instance, vowel marks partially overlap some of the
letters, e.g. the kasrah sign overlaps the two dots of the letter yaa.

3) vowel marks are not always in synch with the main text, so that
they sometimes appear in a displaced position (e.g. on the previous
letter). This is totally unacceptable, of course.

Bottomline: there are multiple problems with the visual rendering of
the diacritics.

bug#10895: Screenshot #1

[sergei karhof]

[-- Attachment #1: Type: text/plain, Size: 24 bytes --]

See attached screenshot

[-- Attachment #2: Screenshot 1.jpg --]
[-- Type: image/jpeg, Size: 161596 bytes --]

bug#10895: Screenshot #1

[Eli Zaretskii]

> Date: Mon, 27 Feb 2012 11:15:41 +0100
> From: sergei karhof <karhof21@gmail.com>
> 
> See attached screenshot

Thanks.  Can you tell which one(s) of these 3 screenshots show correct
display?

bug#10895: Screenshot #2

[sergei karhof]

[-- Attachment #1: Type: text/plain, Size: 1 bytes --]



[-- Attachment #2: Screenshot 2.jpg --]
[-- Type: image/jpeg, Size: 166197 bytes --]

bug#10895: Screenshot #3

[sergei karhof]

[-- Attachment #1: Type: text/plain, Size: 1 bytes --]



[-- Attachment #2: Screenshot 3.jpg --]
[-- Type: image/jpeg, Size: 158913 bytes --]

bug#10895: Quirky behaviours with Arabic text

[Eli Zaretskii]

> Date: Mon, 27 Feb 2012 00:47:36 +0100
> From: sergei karhof <karhof21@gmail.com>
> 
> I want to report some quirky behaviours of Emacs (devel 24.0.93 version,
> running under Windows 7), when dealing with Arabic text:

Thanks.

Jason and Kenichi, could you please take a look at this problem?  The
first question I'd ask is whether this is specific to Windows, or do
we have similar problems on GNU/Linux with libotf?

Sergei, could you please try invoking Emacs with the
`-xrm Emacs.fontBackend:gdi' command-line argument, and see if the
problem persists with the GDI backend?  That's assuming that GDI at
all supports Arabic shaping; I don't know enough about this to tell,
sorry.

bug#10895: Quirky behaviours with Arabic text

[sergei karhof]

[-- Attachment #1: Type: text/plain, Size: 463 bytes --]

On Mon, Feb 27, 2012 at 7:17 PM, Eli Zaretskii <eliz@gnu.org> wrote:

> Sergei, could you please try invoking Emacs with the
> `-xrm Emacs.fontBackend:gdi' command-line argument, and see if the
> problem persists with the GDI backend?  That's assuming that GDI at
> all supports Arabic shaping; I don't know enough about this to tell,
> sorry.

I have just tried this, but it totally messes up the text, as you can
see from screenshot #5 here attached.

[-- Attachment #2: #5.jpg --]
[-- Type: image/jpeg, Size: 70976 bytes --]

bug#10895: Quirky behaviours with Arabic text

[Jason Rumney]

sergei karhof <karhof21@gmail.com> writes:

> On Mon, Feb 27, 2012 at 7:17 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>
>> Sergei, could you please try invoking Emacs with the
>> `-xrm Emacs.fontBackend:gdi' command-line argument, and see if the
>> problem persists with the GDI backend?  That's assuming that GDI at
>> all supports Arabic shaping; I don't know enough about this to tell,
>> sorry.
>
> I have just tried this, but it totally messes up the text, as you can
> see from screenshot #5 here attached.

I would expect that. The GDI font backend has no support for font
shaping, which is definitely required for proper Arabic support.  What I
suspect is that Emacs is doing more in terms of supporting Arabic than
the uniscribe shaping engine expects.

bug#10895: Quirky behaviours with Arabic text

[Eli Zaretskii]

> From: Jason Rumney <jasonr@gnu.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Kenichi Handa <handa@m17n.org>,  10895@debbugs.gnu.org
> Date: Wed, 29 Feb 2012 22:54:41 +0800
> 
> What I suspect is that Emacs is doing more in terms of supporting
> Arabic than the uniscribe shaping engine expects.

Like what, for example?

Does anyone know of a good treatise for Arabic shaping with Uniscribe?

bug#10895: Quirky behaviours with Arabic text

[sergei karhof]

>> What I suspect is that Emacs is doing more in terms of supporting
>> Arabic than the uniscribe shaping engine expects.
>
> Like what, for example?
>
> Does anyone know of a good treatise for Arabic shaping with Uniscribe?

I don't know much about these technical matters (I am just an
end-user), but I would like to point out that as for 'shaping' the
letters (if I understand correctly what it means), it is no issue at
all, because the letters AND the vowel marks are shaped correctly. The
problem is that the vowel marks are not displayed where they should,
that they are out of synch, and that there is not enough vertical
space to contain all the vowel marks.
So I guess that the underlying rendering engine might not be the
problem; maybe it's just the way the whole thing is implemented.

sergei

bug#10895: Quirky behaviours with Arabic text

[Jason Rumney]

sergei karhof <karhof21@gmail.com> writes:

> I don't know much about these technical matters (I am just an
> end-user), but I would like to point out that as for 'shaping' the
> letters (if I understand correctly what it means), it is no issue at
> all, because the letters AND the vowel marks are shaped correctly. The
> problem is that the vowel marks are not displayed where they should,
> that they are out of synch, and that there is not enough vertical
> space to contain all the vowel marks.

Shaping in this context is about shaping whole words, not individual
letters. Positioning vowel marks correctly is part of that.

bug#10895: Bug: vowel marks in Arabic

[sergei karhof]

[-- Attachment #1: Type: text/plain, Size: 1608 bytes --]

Here are two more screenshots:

#4a shows a text exactly as it was loaded from the file. The vowel
marks are totally missing, except for the word 'Allah', which for some
reason displays the vowel marks. The shaddah mark above that word, in
fact, is displayed twice, which is an error! You can see the word
'Allah' which is right at the left of the cursor. Those two marks
(shaped like the number 3, but rotated 180° degrees) above the word
are the shaddah. The reason for this anomaly might be that is some
systems the word Allah is a glyph on its own, which includes the
shaddah mark; if that was the case, this part of the problem could be
solved by removing any additional shaddah mark that might have been
added to that glyph. Anyhow, this was a digression.

The main problem is that the vowel marks are not displayed at all in
the other words.

#4b shows the same text, after the cursor has passed over them. This
is the nearest it comes to displaying the vowel marks correctly. But
again, they are out of synch with the letters, so the whole
vocalization is wrong!

Yet another two problems that I just noticed:
* the vertical space allotted to each line of text is not enough. You
can see from #4b that part of the vowel marks is cut off and thus
missing. This problem might be solved by increasing the inter-line
space, but wouldn't be nice if the proper inter-line space was
calculated and set automatically by the system?
* in #4b you can see, about 1cm at the left of the cursor, that two
vowel marks are overlapping in the same space, and of course they
shouldn't.

[-- Attachment #2: #4a.jpg --]
[-- Type: image/jpeg, Size: 118543 bytes --]

[-- Attachment #3: #4b.jpg --]
[-- Type: image/jpeg, Size: 131239 bytes --]

bug#10895: Quirky behaviours with Arabic text

[Stefan Kangas]

sergei karhof <karhof21@gmail.com> writes:

> I want to report some quirky behaviours of Emacs (devel 24.0.93 version,
> running under Windows 7), when dealing with Arabic text:
>
> 1) in some fonts (e.g. courier New), the vowel marks (harakaat) are
> not displayed, despite the fact that the texts contains them. However,
> if the cursor passes over the relevant text, moving in the
> right-to-left direction, the vowel marks become visible. Curiously, if
> the cursor passes back on the text, moving in the left-to-right
> direction, the vowel marks disappear.
> This problem manifests itself in different ways according to the font used.
> With other fonts, the problem is only partial: only part of the
> diacritic signs is visible, and by passing over them with the cursor
> (from right to left), they become fully visible. When passing over the
> text in the opposite direction, they again become half-visible.
>
> 2) the rendering of the vowel marks is poor even when they are
> displayed. For instance, vowel marks partially overlap some of the
> letters, e.g. the kasrah sign overlaps the two dots of the letter yaa.
>
> 3) vowel marks are not always in synch with the main text, so that
> they sometimes appear in a displaced position (e.g. on the previous
> letter). This is totally unacceptable, of course.
>
> Bottomline: there are multiple problems with the visual rendering of
> the diacritics.

(That was 8 years ago.)

Do you still see this on a recent version of Emacs, such as the recently
released version 27.1?

Would this have been fixed by the recent addition of harfbuzz support?

Best regards,
Stefan Kangas

bug#10895: Quirky behaviours with Arabic text

[Eli Zaretskii]

> From: Stefan Kangas <stefan@marxist.se>
> Date: Mon, 17 Aug 2020 22:34:35 +0000
> Cc: 10895@debbugs.gnu.org
> 
> > 1) in some fonts (e.g. courier New), the vowel marks (harakaat) are
> > not displayed, despite the fact that the texts contains them. However,
> > if the cursor passes over the relevant text, moving in the
> > right-to-left direction, the vowel marks become visible. Curiously, if
> > the cursor passes back on the text, moving in the left-to-right
> > direction, the vowel marks disappear.
> > This problem manifests itself in different ways according to the font used.
> > With other fonts, the problem is only partial: only part of the
> > diacritic signs is visible, and by passing over them with the cursor
> > (from right to left), they become fully visible. When passing over the
> > text in the opposite direction, they again become half-visible.
> >
> > 2) the rendering of the vowel marks is poor even when they are
> > displayed. For instance, vowel marks partially overlap some of the
> > letters, e.g. the kasrah sign overlaps the two dots of the letter yaa.
> >
> > 3) vowel marks are not always in synch with the main text, so that
> > they sometimes appear in a displaced position (e.g. on the previous
> > letter). This is totally unacceptable, of course.
> >
> > Bottomline: there are multiple problems with the visual rendering of
> > the diacritics.
> 
> (That was 8 years ago.)
> 
> Do you still see this on a recent version of Emacs, such as the recently
> released version 27.1?
> 
> Would this have been fixed by the recent addition of harfbuzz support?

I think those problems were fixed long ago, but let's wait for the OP
to confirm, if possible.

bug#10895: Quirky behaviours with Arabic text

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

>> Would this have been fixed by the recent addition of harfbuzz support?
>
> I think those problems were fixed long ago, but let's wait for the OP
> to confirm, if possible.

No further updates within 6 weeks, so I guess we will have to assume
this has been fixed.  I'm therefore closing this bug now.