Re: Re: Arabic support

Re: Re: Arabic support

[Amit Aronovitch]

[-- Attachment #1.1: Type: text/plain, Size: 3718 bytes --]

On Fri, Aug 27, 2010 at 12:56 PM, Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Kenichi Handa <handa@m17n.org>
> > Date: Thu, 26 Aug 2010 10:10:05 +0900
> >
> > I've just committed changes to trunk for Arabic shaping.  If
> > there're any Arabic users in this list, please check the
> > displaying of Arabic text.  On GNU/Linux system, you must
> > compile Emacs with libotf and m17n-lib (configure script
> > should detect them automatically).
>
> Thanks.  However, today's build behaves very strangely in a GUI
> session on MS-Windows.  For starters, cursor motion seems to jump
> across many characters in the "Arabic" line of etc/HELLO.  For
> example, typing C-f in that line, I first move one character at a time
> across "Arabic", as expected, then the cursor jumps to the right paren
> of the leftmost parenthesized part, again as expected, and then I see
> the following strange behavior:
>
>  . C-f moves one character to the left, to buffer position 758, as
>    expected.
>
>  . the next C-f jumps across many characters on the screen and lands
>    on position 764.
>
>  . another C-f jumps to what is reported as position 765, but on the
>    screen those are several characters, maybe 5 or 6.
>
>  . another C-f moves to the left paren at position 766, as expected.
>
>  . yet another C-f moves to position 767, but on the screen the
>    cursor jumps back into one of the characters it jumped across when
>    it landed on position 765 two C-f keypresses earlier.
>
>  . if I type C-b 4 times from this point, I enter a "trap", whereby
>    typing C-b jumps between two characters, whose buffer positions
>    are 764 and 765.  The only way to get out of the trap is with C-a
>    or C-e or C-f.
>
> I don't read Arabic, so I cannot really say whether any of this is
> expected behavior.  (The "trap" with C-b is certainly not the expected
> behavior.)  Do you see anything similar on X?
>
>
1) I confirm that Arabic shaping seems to work fine on my build (27/8/10
rev. 101200, on Linux+X (Debian unstable)).

2) Logical movement with C-f/C-b in the hello file seems fine (I do not see
the trap described above).

3) My Arabic is very basic, and I am not familiar with Arabic computing
(keyboards etc.) - I noticed the following points, but I am not sure what is
the expected behavior (I can only compare to other programs - gedit in this
case):

  a) Column numbers (column-number-mode) behave strangely (I suspect that
m17n-lib's invisible markup consume column numbers). For example as you move
using C-f in the word "هذا" column numbers go through "0,1,4,5" (i.e. the
second character takes up 3 columns). If I change that to "بهذا", the column
positions are "0,1,4,6,7" (the second and third chars take up 3 and 2
columns resp.?).
  In gedit column positions are 1 character per column and do not depend on
the shaping.

  b) Arabic keyboard has the ligature "Lam-Alef" (U+FEFB) on the key marked
"B" in qwerty keyboards. When I type this in emacs, I get Lam and Alef
(which are auto-shaped correctly as the proper ligature). C-d when cursor is
on the ligature erases the Alef and another C-d erases the Lam. This seems
like proper behavior to me. However, in gedit, the "B" key produces a
(U+FEFB) which is always displayed as a ligature, deleted in a single Del
press, and never connected to previous character. Cut and pasting this into
emacs, I get a similar behavior there.
The question is: do Arabic users expect to be able to produce this "stiff"
ligature? Is the behavior of gedit a bug? Should the emacs "Lam-Alef" key
behave as it does (i.e. produce two characters)?

   thanks,
      Amit Aronovitch

[-- Attachment #1.2: Type: text/html, Size: 4481 bytes --]

[-- Attachment #2: Type: text/plain, Size: 140 bytes --]

_______________________________________________
emacs-bidi mailing list
emacs-bidi@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-bidi

Re: Re: Arabic support

[James Cloos]

>>>>> "AA" == Amit Aronovitch <aronovitch@gmail.com> writes:

AA>  b) Arabic keyboard has the ligature "Lam-Alef" (U+FEFB) on the key marked
AA> "B" in qwerty keyboards. When I type this in emacs, I get Lam and Alef
AA> (which are auto-shaped correctly as the proper ligature).

Emacs' Arabic keyboard is based on a patch I proposed, which in turn is
based on typing the relevant keys with the X11 keyboard set to the Arabic
keyboard from xkeyboard-config.  The mapping of U+FEFB, U+FEF9, U+FEF7
and U+FEF5 to the strings "لا", "لإ", "لأ", "لآ" (I hope those pasted
correctly) come from a patch to libX11's utf-8 Compose file which I pushed 
and which was written and submitted by Khaled Hosny, who has been quite
active

Re: Re: Arabic support

[James Cloos]

>>>>> "AA" == Amit Aronovitch <aronovitch@gmail.com> writes:

AA>  b) Arabic keyboard has the ligature "Lam-Alef" (U+FEFB) on the key marked
AA> "B" in qwerty keyboards. When I type this in emacs, I get Lam and Alef
AA> (which are auto-shaped correctly as the proper ligature).

Emacs' Arabic keyboard is based on a patch I proposed, which in turn is
based on typing the relevant keys with the X11 keyboard set to the
Arabic keyboard from xkeyboard-config.  The mapping of U+FEFB, U+FEF9,
U+FEF7 and U+FEF5 to the strings "لا", "لإ", "لأ", "لآ" (I hope those
pasted correctly) come from a patch to libX11's utf-8 Compose file which
I pushed and which was submitted by Khaled Hosny, who has been quite
active in i18n circles.

My guess is that the behaviour you described, given why it occurs, is
the desired behaviour.

But that is only a guess.  I don't read or speak any of the languages
which use the Arabic script, I'm just interested in i18n and pan-
language typography and font design.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: Re: Arabic support

[Kenichi Handa]

In article <AANLkTinFrEnuW=oPeBqg6=wYegbrR+Lani2WcmDYstVO@mail.gmail.com>, Amit Aronovitch <aronovitch@gmail.com> writes:

> 1) I confirm that Arabic shaping seems to work fine on my build (27/8/10
> rev. 101200, on Linux+X (Debian unstable)).

> 2) Logical movement with C-f/C-b in the hello file seems fine (I do not see
> the trap described above).

Thank yor for testing them.

> 3) My Arabic is very basic, and I am not familiar with Arabic computing
> (keyboards etc.) - I noticed the following points, but I am not sure what i=
> s
> the expected behavior (I can only compare to other programs - gedit in this
> case):

>   a) Column numbers (column-number-mode) behave strangely (I suspect that
> m17n-lib's invisible markup consume column numbers). For example as you mov=
> e
> using C-f in the word "=D9=87=D8=B0=D8=A7" column numbers go through "0,1,4=
> ,5" (i.e. the
> second character takes up 3 columns). If I change that to "=D8=A8=D9=87=D8=
> =B0=D8=A7", the column
> positions are "0,1,4,6,7" (the second and third chars take up 3 and 2
> columns resp.?).
>   In gedit column positions are 1 character per column and do not depend on
> the shaping.

I've just committed a fix for this bug.  It's not related to
m17n-lib.

---
Kenichi Handa
handa@m17n.org

Re: Re: Arabic support

[Amit Aronovitch]

[-- Attachment #1.1: Type: text/plain, Size: 1990 bytes --]

On Mon, Aug 30, 2010 at 5:07 AM, Kenichi Handa <handa@m17n.org> wrote:

> In article <AANLkTinFrEnuW=oPeBqg6=wYegbrR+Lani2WcmDYstVO@mail.gmail.com<wYegbrR%2BLani2WcmDYstVO@mail.gmail.com>>,
> Amit Aronovitch <aronovitch@gmail.com> writes:
>
> > 1) I confirm that Arabic shaping seems to work fine on my build (27/8/10
> > rev. 101200, on Linux+X (Debian unstable)).
>
> > 2) Logical movement with C-f/C-b in the hello file seems fine (I do not
> see
> > the trap described above).
>
> Thank yor for testing them.
>
> > 3) My Arabic is very basic, and I am not familiar with Arabic computing
> > (keyboards etc.) - I noticed the following points, but I am not sure what
> i=
> > s
> > the expected behavior (I can only compare to other programs - gedit in
> this
> > case):
>
> >   a) Column numbers (column-number-mode) behave strangely (I suspect that
> > m17n-lib's invisible markup consume column numbers). For example as you
> mov=
> > e
> > using C-f in the word "=D9=87=D8=B0=D8=A7" column numbers go through
> "0,1,4=
> > ,5" (i.e. the
> > second character takes up 3 columns). If I change that to
> "=D8=A8=D9=87=D8=
> > =B0=D8=A7", the column
> > positions are "0,1,4,6,7" (the second and third chars take up 3 and 2
> > columns resp.?).
> >   In gedit column positions are 1 character per column and do not depend
> on
> > the shaping.
>
> I've just committed a fix for this bug.  It's not related to
> m17n-lib.
>
>
Thanks. Much better now :-)

I also checked the diacritics (tashkil): It seems that they do not take up
column number in Emacs.

In gedit, cursor movement is similar, but the vowels there do take up column
number (as for cursor movement, as in emacs: forwards/backwards skips them,
while 'delete' handles them separately). I find this behavior more
consistent with the way both programs handle the lam-alef ligature (one
cursor-movement space, but two column numbers).
However, as I said, I do not know which behavior is the most natural for
Arabic users.

   AA

[-- Attachment #1.2: Type: text/html, Size: 2721 bytes --]

[-- Attachment #2: Type: text/plain, Size: 140 bytes --]

_______________________________________________
emacs-bidi mailing list
emacs-bidi@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-bidi

Re: Re: Arabic support

[mhibti]

Dear all,
I have tried the MS-Windows (XP) version of 30/08.
The hello message is not correclty displayed, it is written from left to right, 
moreover emacs crashes when trying to write in arabic.
I'm very sorry I couldn't do more tests..




----- Mail Original -----
De: "Amit Aronovitch" <aronovitch@gmail.com>
À: "Kenichi Handa" <handa@m17n.org>
Cc: emacs-bidi@gnu.org, emacs-devel@gnu.org
Envoyé: Lundi 30 Août 2010 16h11:06 GMT +01:00 Amsterdam / Berlin / Berne / Rome / Stockholm / Vienne
Objet: Re: [emacs-bidi] Re: Arabic support






On Mon, Aug 30, 2010 at 4:42 PM, Amit Aronovitch < aronovitch@gmail.com > wrote: 








On Mon, Aug 30, 2010 at 5:07 AM, Kenichi Handa < handa@m17n.org > wrote: 



In article <AANLkTinFrEnuW=oPeBqg6= wYegbrR+Lani2WcmDYstVO@mail.gmail.com >, Amit Aronovitch < aronovitch@gmail.com > writes: 

> 1) I confirm that Arabic shaping seems to work fine on my build (27/8/10 
> rev. 101200, on Linux+X (Debian unstable)). 

> 2) Logical movement with C-f/C-b in the hello file seems fine (I do not see 
> the trap described above). 

Thank yor for testing them. 


> 3) My Arabic is very basic, and I am not familiar with Arabic computing 
> (keyboards etc.) - I noticed the following points, but I am not sure what i= 
> s 

> the expected behavior (I can only compare to other programs - gedit in this 
> case): 

> a) Column numbers (column-number-mode) behave strangely (I suspect that 
> m17n-lib's invisible markup consume column numbers). For example as you mov= 
> e 
> using C-f in the word "=D9=87=D8=B0=D8=A7" column numbers go through "0,1,4= 
> ,5" (i.e. the 
> second character takes up 3 columns). If I change that to "=D8=A8=D9=87=D8= 
> =B0=D8=A7", the column 

> positions are "0,1,4,6,7" (the second and third chars take up 3 and 2 
> columns resp.?). 
> In gedit column positions are 1 character per column and do not depend on 
> the shaping. 

I've just committed a fix for this bug. It's not related to 
m17n-lib. 




Thanks. Much better now :-) 


I also checked the diacritics (tashkil): It seems that they do not take up column number in Emacs. 


In gedit, cursor movement is similar, but the vowels there do take up column number (as for cursor movement, as in emacs: forwards/backwards skips them, while 'delete' handles them separately). I find this behavior more consistent with the way both programs handle the lam-alef ligature (one cursor-movement space, but two column numbers). 
However, as I said, I do not know which behavior is the most natural for Arabic users. 




Checking the *Hebrew* diacritics (nikkud), I noticed a problem: 
In some cases the diacritics are displayed in the wrong position (their "real" cursor position is correct, which makes the UI *very* confusing). e.g. if you type "‫ עָלֵינוּ‬" , the Qamatz (first vowel) appears under the space instead of under the Ain (first letter). If you remove the space, the Qamatz does not appear at all. The Zeire (second vowel) appears under the Ain (first vowel) instead of the Lamed (second letter). However, the Shuruk sticks to the Vav (last letter) as it should (though the positioning is too close and to high IMHO). 
I do not know if this issue is specific to my build. 
My complete config.log is available here: 


http://dl.dropbox.com/u/6960989/dumps/config.log 


AA 


_______________________________________________
emacs-bidi mailing list
emacs-bidi@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-bidi

Re: Re: Arabic support

[Eli Zaretskii]

> Date: Tue, 31 Aug 2010 03:41:56 +0200 (CEST)
> From: mhibti@free.fr
> Cc: emacs-bidi@gnu.org, Kenichi Handa <handa@m17n.org>, emacs-devel@gnu.org
> 
> I have tried the MS-Windows (XP) version of 30/08.
> The hello message is not correclty displayed, it is written from left to right, 

Thanks for testing.  Could you please send an image of how the current
greeting should be displayed in correct visual order on the screen?
It's possible that the order of the characters in the HELLO file is
incorrect, and that's one reason why it is displayed incorrectly.

Thanks.

Re: Re: Arabic support

[Kenichi Handa]

In article <AANLkTimnZjHOeuQci7UomVux3swgs_ydQUXfnT_cbMMM@mail.gmail.com>, Amit Aronovitch <aronovitch@gmail.com> writes:

> I also checked the diacritics (tashkil): It seems that they do not take up
> column number in Emacs.

> In gedit, cursor movement is similar, but the vowels there do take up column
> number (as for cursor movement, as in emacs: forwards/backwards skips them,
> while 'delete' handles them separately). I find this behavior more
> consistent with the way both programs handle the lam-alef ligature (one
> cursor-movement space, but two column numbers).
> However, as I said, I do not know which behavior is the most natural for
> Arabic users.

In Emacs, the column number affects when a user types C-n
(or C-p) to go to (roughly) the same x-position of the next
(or previous) line.  So, the column number should reflects
the x-position, and for that, zero-width combining
characters should not be counted into the column number.  Of
course, when a text is displayed by a variable-pitch font,
it is not good to use the column number for such a purpose,
but that is what Emacs has been done for long.

The lam-aref ligature case should be fixed so that the
ligature glyph is counted as one-column.

It seems that gedit uses the actual x-pixel-position to move
the cursor down.  It will be better Emacs does the same in
the future.

---
Kenichi Handa
handa@m17n.org

Re: Re: Arabic support

["Martin J. Dürst"]

We have made similar observations with what might be double reordering 
(or no reordering) on a Windows system. I expect we will report more 
details tomorrow.

Regards,    Martin.

On 2010/09/01 11:17, Kenichi Handa wrote:
> In article<E1Oq50d-0006YC-8u@fencepost.gnu.org>, Eli Zaretskii<eliz@gnu.org>  writes:
>
>>> In Emacs, bidi reordering is done by Emacs itself, so the `shape'
>>> method of font backend should not reorder glyphs.  But, perhaps
>>> Uniscribe backend reorders Arabic text, right?
>
>> No, not AFAIK.  We call the ScriptItemize API of Uniscribe with NULL
>> as the 4th and 5th arguments, which AFAIU should disable reordering.
>> Perhaps Jason could chime in and tell if I'm right here.
>
> I read the function uniscribe_shape roughly.  It has this
> code:
>
>    for (i = 0; i<  nitems; i++)
>      {
>        int nglyphs, nchars_in_run, rtl = items[i].a.fRTL ? -1 : 1;
> [...]
>            if (SUCCEEDED (result))
> 	    {
> 	      int j, nclusters, from, to;
>
> 	      from = rtl>  0 ? 0 : nchars_in_run - 1;
>
> Doesn't it mean uniscribe_shape reorders glyphs?


-- 
#-# Martin J. Dürst, Professor, Aoyama Gakuin University
#-# http://www.sw.it.aoyama.ac.jp   mailto:duerst@it.aoyama.ac.jp

Re: Re: Arabic support

[Eli Zaretskii]

> From: Kenichi Handa <handa@m17n.org>
> Cc: eliz@gnu.org, emacs-bidi@gnu.org, emacs-devel@gnu.org
> Date: Wed, 01 Sep 2010 11:55:38 +0900
> 
> In Emacs, the column number affects when a user types C-n
> (or C-p) to go to (roughly) the same x-position of the next
> (or previous) line.  So, the column number should reflects
> the x-position, and for that, zero-width combining
> characters should not be counted into the column number.

But does the implementation of current-column, move-to-column and
friends support that?  Perhaps I'm missing something, but my reading
of current_column_1 and its subroutines is that it only supports
display strings, composed characters, and display tables.  Do
zero-width characters use any of these mechanisms?

Re: Re: Arabic support

[Kenichi Handa]

In article <E1OqfPX-0008NB-4f@fencepost.gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> > In Emacs, the column number affects when a user types C-n
> > (or C-p) to go to (roughly) the same x-position of the next
> > (or previous) line.  So, the column number should reflects
> > the x-position, and for that, zero-width combining
> > characters should not be counted into the column number.

> But does the implementation of current-column, move-to-column and
> friends support that?  Perhaps I'm missing something, but my reading
> of current_column_1 and its subroutines is that it only supports
> display strings, composed characters, and display tables.  Do
> zero-width characters use any of these mechanisms?

Those functions basically uses char-width-table to get
width-in-column of each character, and zero-width combining
characters have 0 in that table.

---
Kenichi Handa
handa@m17n.org

RE: Re: Arabic support

[大嶋 俊祐]

[-- Attachment #1: Type: text/plain, Size: 1824 bytes --]


 Hello, everybody.

 This appended picture shows a test of bidi-display-reordering. The upper case is the case that bidi-display-reordering is nil. The lower case is non-nil. The problem is that the characters are ordered left to right even they should been right to left when bidi-display-reordering is non-nil. This test is in Emacs Ver. 24.0.50.1 (i686-pc-cygwin) on Microsoft Windows XP Professional Version 2002 Service Pack 3.

 Shunsuke Oshima ej9460@hotmail.com

> Date: Wed, 1 Sep 2010 12:47:23 +0900
> From: duerst@it.aoyama.ac.jp
> To: handa@m17n.org
> CC: eliz@gnu.org; emacs-bidi@gnu.org; emacs-devel@gnu.org; jasonr@gnu.org
> Subject: Re: [emacs-bidi] Re: Arabic support
> 
> We have made similar observations with what might be double reordering 
> (or no reordering) on a Windows system. I expect we will report more 
> details tomorrow.
> 
> Regards, Martin.
> 
> On 2010/09/01 11:17, Kenichi Handa wrote:
>> In article<E1Oq50d-0006YC-8u@fencepost.gnu.org>, Eli Zaretskii<eliz@gnu.org> writes:
>>
>>>> In Emacs, bidi reordering is done by Emacs itself, so the `shape'
>>>> method of font backend should not reorder glyphs. But, perhaps
>>>> Uniscribe backend reorders Arabic text, right?
>>
>>> No, not AFAIK. We call the ScriptItemize API of Uniscribe with NULL
>>> as the 4th and 5th arguments, which AFAIU should disable reordering.
>>> Perhaps Jason could chime in and tell if I'm right here.
>>
>> I read the function uniscribe_shape roughly. It has this
>> code:
>>
>> for (i = 0; i< nitems; i++)
>> {
>> int nglyphs, nchars_in_run, rtl = items[i].a.fRTL ? -1 : 1;
>> [...]
>> if (SUCCEEDED (result))
>> {
>> int j, nclusters, from, to;
>>
>> from = rtl> 0 ? 0 : nchars_in_run - 1;
>>
>> Doesn't it mean uniscribe_shape reorders glyphs? 		 	   		  

[-- Attachment #2: faith.txt --]
[-- Type: text/plain, Size: 45 bytes --]

after
אני מאמין
אני >> מאמין

[-- Attachment #3: bdr.PNG --]
[-- Type: image/png, Size: 11731 bytes --]

[-- Attachment #4: Type: text/plain, Size: 140 bytes --]

_______________________________________________
emacs-bidi mailing list
emacs-bidi@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-bidi

Re: Re: Arabic support

[Eli Zaretskii]

> From: 大嶋 俊祐 <ej9460@hotmail.com>
> Date: Thu, 2 Sep 2010 16:45:02 +0900
> Cc: emacs-bidi@gnu.org, emacs-devel@gnu.org, jasonr@gnu.org
> 
>  This appended picture shows a test of bidi-display-reordering. The upper case is the case that bidi-display-reordering is nil. The lower case is non-nil. The problem is that the characters are ordered left to right even they should been right to left when bidi-display-reordering is non-nil. This test is in Emacs Ver. 24.0.50.1 (i686-pc-cygwin) on Microsoft Windows XP Professional Version 2002 Service Pack 3.

Thanks.  This is a Cygwin build, which is supposed to use (a port of)
libotf.  It isn't supposed to use Uniscribe directly.

So this particular problem could be the result of one or more of the
following factors:

 . Some problem in the build process (I think Handa-san mentioned some
   special actions for a proper build, like link against specific
   libraries).

 . Some bug in the ported libotf, which screws up bidirectional
   display when the text is already reordered.

 . Some bug in Emacs related to display bidirectional text.

The last one seems extremely unlikely, since AFAIU it works for
Handa-san on GNU/Linux and for me on MS-Windows, with Hebrew text.

To summarize, I don't think this problem is the same one as seen in
the native Windows build.

Re: Re: Arabic support

[Eli Zaretskii]

And another question: AFAIU, an LGSTRING specifies characters as
Unicode codepoints, while the Windows Uniscribe APIs expect wchar_t
wide characters, which on Windows means UTF-16.  This means we should
encode the codepoints in LGSTRINGs to UTF-16 before passing them to
Uniscribe, rather than passing them unaltered, right?  The current
code will break for characters whose Unicode codepoints are beyond the
BMP, right?

Re: Re: Arabic support

["Martin J. Dürst"]

Hello Eli,

Many thanks for your quick and detailled feedback. Any hints on how we 
could go about to figure out which of the factors below is the culprit?
(e.g. check which libraries we linked against,...)

Regards,    Martin.

On 2010/09/02 18:31, Eli Zaretskii wrote:
>> From: 大嶋 俊祐<ej9460@hotmail.com>
>> Date: Thu, 2 Sep 2010 16:45:02 +0900
>> Cc: emacs-bidi@gnu.org, emacs-devel@gnu.org, jasonr@gnu.org
>>
>>   This appended picture shows a test of bidi-display-reordering. The upper case is the case that bidi-display-reordering is nil. The lower case is non-nil. The problem is that the characters are ordered left to right even they should been right to left when bidi-display-reordering is non-nil. This test is in Emacs Ver. 24.0.50.1 (i686-pc-cygwin) on Microsoft Windows XP Professional Version 2002 Service Pack 3.
>
> Thanks.  This is a Cygwin build, which is supposed to use (a port of)
> libotf.  It isn't supposed to use Uniscribe directly.
>
> So this particular problem could be the result of one or more of the
> following factors:
>
>   . Some problem in the build process (I think Handa-san mentioned some
>     special actions for a proper build, like link against specific
>     libraries).
>
>   . Some bug in the ported libotf, which screws up bidirectional
>     display when the text is already reordered.
>
>   . Some bug in Emacs related to display bidirectional text.
>
> The last one seems extremely unlikely, since AFAIU it works for
> Handa-san on GNU/Linux and for me on MS-Windows, with Hebrew text.
>
> To summarize, I don't think this problem is the same one as seen in
> the native Windows build.
>
>

-- 
#-# Martin J. Dürst, Professor, Aoyama Gakuin University
#-# http://www.sw.it.aoyama.ac.jp   mailto:duerst@it.aoyama.ac.jp

Re: Re: Arabic support

[Eli Zaretskii]

> From: Jason Rumney <jasonr@gnu.org>
> Cc: handa@m17n.org,  emacs-bidi@gnu.org,  emacs-devel@gnu.org
> Date: Thu, 02 Sep 2010 21:09:57 +0800
> 
> In practice I don't think any of the scripts that we use the shaping
> engine for lie beyond the BMP.

By "scripts that we use" you mean scripts supported by Uniscribe?
scripts supported by Emacs? scripts available on most Windows systems?
something else?

I do see some useful characters beyond the BMP, e.g. the 1DXXX block
for mathematical symbols, 1F1XX for parenthesized and circled Latin
letters, CJK Ideographs extensions in the 20XXX and 2AXXX ranges, etc.

Thanks.

Re: Re: Arabic support

[Kenichi Handa]

In article <jwvk4n6aqmk.fsf-monnier+emacs@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca> writes:

> > It seems that gedit uses the actual x-pixel-position to move the
> > cursor down.  It will be better Emacs does the same in the future.

> The future has passed when Emacs-23 was released.

Wow, I didn't notice that.

---
Kenichi Handa
handa@m17n.org

Re: Re: Arabic support

[Kenichi Handa]

In article <AANLkTinO+-rABef7x7Et1y14LbePwJ4LbduVQMeT66SK@mail.gmail.com>, Amit Aronovitch <aronovitch@gmail.com> writes:

> Checking the  *Hebrew* diacritics (nikkud), I noticed a problem:
> In some cases the diacritics are displayed in the wrong position (their
> "real" cursor position is correct, which makes the UI *very* confusing).
> e.g. if you type "=E2=80=AB =D7=A2=D6=B8=D7=9C=D6=B5=D7=99=D7=A0=D7=95=D6=
> =BC=E2=80=AC" , the Qamatz (first vowel) appears under the
> space instead of under the Ain (first letter). If you remove the space, the
> Qamatz does not appear at all. The Zeire (second vowel) appears under the
> Ain (first vowel) instead of the Lamed (second letter). However, the Shuruk
> sticks to the Vav (last letter) as it should (though the positioning is too
> close and to high IMHO).
> I do not know if this issue is specific to my build.

I think it is specific to your Hebrew font.  Please tell me
which font is selected by typing C-u C-x = on some Hebrew
character.

---
Kenichi Handa
handa@m17n.org

Re: Re: Arabic support

[Amit Aronovitch]

[-- Attachment #1.1: Type: text/plain, Size: 1402 bytes --]

On Fri, Sep 3, 2010 at 4:25 PM, Eli Zaretskii <eliz@gnu.org> wrote:

<- Snipped text ->


> Even if everything I said above is correct, there are complications.
> ABCDE could be inside an embedding with left to right override, like
> this:
>
>  foobar RLO ABCDE PDF
>
> This should be displayed as
>
>  foobar ABCDE
>
> i.e., "ABCDE" is not reordered, but displayed in the logical order, as
> forced by RLO.  Therefore, the reshaped "XYZ" should also be displayed
> left to right:
>
>  foobar XYZ
>
> But, if I understand correctly how composition works, the
> auto-composed sequence in this case will still be just "XYZ", without
> the RLO and PDF control characters.  So the `shape' method of the font
> driver will still see just "XYZ" in the LGSTRING, without the control
> characters, and will reorder "XYZ", which is incorrect.
>
> If we need the `shape' method to reorder glyphs, then in order for it
> do its job correctly, we need to give it the entire bidi context of
> the string we are asking it to reshape.  In the above example, we need
> to tell it about the override directive, i.e. pass it "ABCDE" with
> surrounding RLO and PDF controls.  This flies in the face of the
> current design, which separates reordering from glyph shaping.
>
>
Is that a typo? I think you mean LRO (U+202D) rather than RLO in all cases
above.
(Just commenting, to avoid further misunderstandings).

   Amit

[-- Attachment #1.2: Type: text/html, Size: 1947 bytes --]

[-- Attachment #2: Type: text/plain, Size: 140 bytes --]

_______________________________________________
emacs-bidi mailing list
emacs-bidi@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-bidi

Re: Re: Arabic support

[Eli Zaretskii]

> Date: Fri, 3 Sep 2010 17:32:33 +0300
> From: Amit Aronovitch <aronovitch@gmail.com>
> Cc: Kenichi Handa <handa@m17n.org>, emacs-bidi@gnu.org, emacs-devel@gnu.org, jasonr@gnu.org
> 
> Is that a typo? I think you mean LRO (U+202D) rather than RLO in all cases
> above.

Yes, sorry.  I meant LRO.

Re: Re: Arabic support

[mhibti]

[-- Attachment #1: Type: text/plain, Size: 2688 bytes --]

I have tested the binaries on XP windows and here is some points:

- The hello message is Ok.
- I tried some simple arabic text and it seems working as I expected at least for me (I'm not using tashkeel).
- The second remark of Mahmoud is not an issue for me see attached picture (coupure).
- I have problems with copy/paste arabic text but I guess this may be a coding issue.

Best regards,
Mohamed


----- Mail Original -----
De: "Thamer Mahmoud" <thamer.mahmoud@gmail.com>
À: emacs-bidi@gnu.org
Cc: emacs-devel@gnu.org
Envoyé: Lundi 6 Septembre 2010 15h45:00 GMT +01:00 Amsterdam / Berlin / Berne / Rome / Stockholm / Vienne
Objet: [emacs-bidi] Re: Arabic support

>> From: Kenichi Handa <handa@m17n.org>
>> Date: Thu, 26 Aug 2010 10:10:05 +0900
>> 
>> I've just committed changes to trunk for Arabic shaping.  If
>> there're any Arabic users in this list, please check the
>> displaying of Arabic text.  On GNU/Linux system, you must
>> compile Emacs with libotf and m17n-lib (configure script
>> should detect them automatically).
>

Thanks for working on this. Here is my take:

* Attached are two screenshots showing the Arabic line from the HELLO
  file rendered by gedit and Emacs using the same font (Nazli-20 from
  ttf-farsiweb). Notice that in Emacs not all fonts have their LAM and
  ALIF properly replaced by the LAM-ALIF ligature. Also the diacritics
  (SHADDA) appears lower and less legible for the same font.

* The third attachment shows that when highlighting a region of an
  Arabic word, the cursor at the edges of the visible selection "breaks"
  the shaping and reshapes the characters around it into their isolated
  form. This creates a wave-effect of moving characters with some
  visible artifacts and bad indention issues.

* While the cursor is at a composed character (e.g., SEEN+SHADDA),
  pressing C-p moves point unexpectedly to the beginning of the current
  line.

* I do at least see one "trap" with C-p, although it is hard to
  reproduce. You can try moving 4 or 5 lines below the Arabic line in
  the HELLO file, then move upward using 4-5 C-p and get the cursor at
  the SEEN+SHADDA. After which any further C-p jumps between SEEN and
  LAM-ALIF, never going to the previous line.

* For those using Debian (Squeeze), I had to install not just the
  libm17n and libm17n-dev packages, but also m17n-db. It seems that the
  configure script doesn't detect or know about the status of (the
  Debian-specific) m17n-db.

Thanks again,
Thamer


_______________________________________________
emacs-bidi mailing list
emacs-bidi@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-bidi

[-- Attachment #2: coupure.jpg --]
[-- Type: image/jpeg, Size: 9099 bytes --]

[-- Attachment #3: hello.jpg --]
[-- Type: image/jpeg, Size: 57496 bytes --]

[-- Attachment #4: Type: text/plain, Size: 140 bytes --]

_______________________________________________
emacs-bidi mailing list
emacs-bidi@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-bidi

Re: Re: Arabic support

[Eli Zaretskii]

> Date: Tue, 7 Sep 2010 02:59:48 +0200 (CEST)
> From: mhibti@free.fr
> Cc: emacs-bidi@gnu.org, emacs-devel@gnu.org
> 
> - I have problems with copy/paste arabic text but I guess this may be a coding issue.

Thanks for testing.

Would you please describe the problems you have with copy/paste?
There should be no problems with encoding on Windows, as Emacs uses
UTF-16 automatically to copy/paste text on Windows.

Re: Re: Arabic support

[mhibti]

[-- Attachment #1: Type: text/plain, Size: 979 bytes --]

Thanks Eli for all your efforts !

It is simple. When copying  a text in arabic (see pictures) and pasting in the emacs buffer,
the result is a series of "?" characters (see resu.jpg).
I tried decode-coding-region (with utf-8, utf-16, and latin-1) without succes.


----- Mail Original -----
De: "Eli Zaretskii" <eliz@gnu.org>
À: mhibti@free.fr
Cc: emacs-bidi@gnu.org, emacs-devel@gnu.org
Envoyé: Mardi 7 Septembre 2010 05h03:07 GMT +01:00 Amsterdam / Berlin / Berne / Rome / Stockholm / Vienne
Objet: Re: [emacs-bidi] Re: Arabic support

> Date: Tue, 7 Sep 2010 02:59:48 +0200 (CEST)
> From: mhibti@free.fr
> Cc: emacs-bidi@gnu.org, emacs-devel@gnu.org
> 
> - I have problems with copy/paste arabic text but I guess this may be a coding issue.

Thanks for testing.

Would you please describe the problems you have with copy/paste?
There should be no problems with encoding on Windows, as Emacs uses
UTF-16 automatically to copy/paste text on Windows.

[-- Attachment #2: emacs-transla.jpg --]
[-- Type: image/jpeg, Size: 49767 bytes --]

[-- Attachment #3: resu.jpg --]
[-- Type: image/jpeg, Size: 97582 bytes --]

[-- Attachment #4: Type: text/plain, Size: 140 bytes --]

_______________________________________________
emacs-bidi mailing list
emacs-bidi@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-bidi

Re: Re: Arabic support

[TAKAHASHI Naoto]

Thamer Mahmoud writes:

> * Attached are two screenshots showing the Arabic line from the HELLO
>   file rendered by gedit and Emacs using the same font (Nazli-20 from
>   ttf-farsiweb). Notice that in Emacs not all fonts have their LAM and
>   ALIF properly replaced by the LAM-ALIF ligature. Also the diacritics
>   (SHADDA) appears lower and less legible for the same font.

This problem is caused by the combination of nazli.ttf, which has
rather unusual OTF tables, and a bug in m17n-db.  I will try to find a
workaround.

-- 
TAKAHASHI Naoto
ntakahas@m17n.org

Re: Re: Arabic support

[Eli Zaretskii]

> Date: Tue, 7 Sep 2010 05:34:04 +0200 (CEST)
> From: mhibti@free.fr
> Cc: emacs-bidi@gnu.org, emacs-devel@gnu.org
> 
> It is simple. When copying a text in arabic (see pictures) and
> pasting in the emacs buffer, the result is a series of "?"
> characters (see resu.jpg).

From which application did you copy the Arabic text?
Do you see the same problems with other applications, or just with
this one?
Does it work to copy/paste from one Emacs instance to another?  (I
mean actually invoking "emacs -Q" twice, not pasting from one Emacs
frame to another in the same session.)
Does it work to copy FROM Emacs to another application that is not
Emacs?

Re: Re: Arabic support

[mhibti]

You are right !

After invoking emacs -q I could do the following :

- copy from emacs to another application (it worked).
- copy arabic text from two different applications to emacs 
it works correctly exepted that tashkeel seems lost when the source include it.
But after verification if I try to mark the region in question the tashkeel appears :)

In my dot emacs i found what may be the cause of my problem.


 '(selection-coding-system (quote utf-8-dos))
 '(unify-8859-on-decoding-mode t)
 '(unify-8859-on-encoding-mode t)

Thanks



----- Mail Original -----
De: "Eli Zaretskii" <eliz@gnu.org>
À: mhibti@free.fr
Cc: emacs-bidi@gnu.org, emacs-devel@gnu.org
Envoyé: Mardi 7 Septembre 2010 06h39:53 GMT +01:00 Amsterdam / Berlin / Berne / Rome / Stockholm / Vienne
Objet: Re: [emacs-bidi] Re: Arabic support

> Date: Tue, 7 Sep 2010 05:34:04 +0200 (CEST)
> From: mhibti@free.fr
> Cc: emacs-bidi@gnu.org, emacs-devel@gnu.org
> 
> It is simple. When copying a text in arabic (see pictures) and
> pasting in the emacs buffer, the result is a series of "?"
> characters (see resu.jpg).

From which application did you copy the Arabic text?
Do you see the same problems with other applications, or just with
this one?
Does it work to copy/paste from one Emacs instance to another?  (I
mean actually invoking "emacs -Q" twice, not pasting from one Emacs
frame to another in the same session.)
Does it work to copy FROM Emacs to another application that is not
Emacs?

Re: Re: Arabic support

[Eli Zaretskii]

> Date: Tue, 7 Sep 2010 17:08:01 +0200 (CEST)
> From: mhibti@free.fr
> Cc: emacs-bidi@gnu.org, emacs-devel@gnu.org
> 
> - copy arabic text from two different applications to emacs 
> it works correctly exepted that tashkeel seems lost when the source include it.
> But after verification if I try to mark the region in question the tashkeel appears :)

It's probably some bad interaction between compositions and the
handling of faces in the bidirectional display.  Perhaps Handa-san
could take a look at xdisp.c:handle_stop_backwards and how it is
called inside next_element_from_buffer -- there might be some bugs
there whereby only part of the composed sequence is redrawn when the
region is extended or contracted.

> In my dot emacs i found what may be the cause of my problem.
> 
> 
>  '(selection-coding-system (quote utf-8-dos))

This one is your problem: you should never do that on MS-Windows.

>  '(unify-8859-on-decoding-mode t)
>  '(unify-8859-on-encoding-mode t)

These are obsolete, as everything is always unified in Emacs 24.

Re: Re: Arabic support

[Kenichi Handa]

In article <83eicy5epd.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> It's probably some bad interaction between compositions and the
> handling of faces in the bidirectional display.

I found that the problem is that the current composition for
Arabic requires that a whole word must be composed.  So, if
there's a face change within a word, Arabic composition
function is given just a partial word, and that results in
the incorrect Arabic shaping.  This is a difficult problem,
and I need a time to find a solution.

---
Kenichi Handa
handa@m17n.org

Re: Re: Arabic support

[Kenichi Handa]

In article <tl77himmofd.fsf@m17n.org>, Kenichi Handa <handa@m17n.org> writes:

> In article <83eicy5epd.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:
> > It's probably some bad interaction between compositions and the
> > handling of faces in the bidirectional display.

> I found that the problem is that the current composition for
> Arabic requires that a whole word must be composed.  So, if
> there's a face change within a word, Arabic composition
> function is given just a partial word, and that results in
> the incorrect Arabic shaping.  This is a difficult problem,
> and I need a time to find a solution.

I've just installed a fix to trunk.

---
Kenichi Handa
handa@m17n.org

Re: Re: Arabic support

[Eli Zaretskii]

> From: Kenichi Handa <handa@m17n.org>
> Cc: eliz@gnu.org, emacs-bidi@gnu.org, mhibti@free.fr, emacs-devel@gnu.org
> Date: Wed, 22 Sep 2010 12:54:26 +0900
> 
> In article <tl77himmofd.fsf@m17n.org>, Kenichi Handa <handa@m17n.org> writes:
> 
> > In article <83eicy5epd.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:
> > > It's probably some bad interaction between compositions and the
> > > handling of faces in the bidirectional display.
> 
> > I found that the problem is that the current composition for
> > Arabic requires that a whole word must be composed.  So, if
> > there's a face change within a word, Arabic composition
> > function is given just a partial word, and that results in
> > the incorrect Arabic shaping.  This is a difficult problem,
> > and I need a time to find a solution.
> 
> I've just installed a fix to trunk.

Thanks.

Re: Arabic support

[Thamer Mahmoud]

[-- Attachment #1: Type: text/plain, Size: 930 bytes --]

Kenichi Handa <handa@m17n.org> writes:
>> I found that the problem is that the current composition for
>> Arabic requires that a whole word must be composed.  So, if
>> there's a face change within a word, Arabic composition
>> function is given just a partial word, and that results in
>> the incorrect Arabic shaping.  This is a difficult problem,
>> and I need a time to find a solution.
>
> I've just installed a fix to trunk.

I can confirm that the issue with unshaped glyphs while highlighting
words is now fixed.  Thanks.

However, long Arabic strings still have unshaped middle parts and bad
margin.  See the attached screenshot which is the output of M-30-<BAA>
in an empty buffer.

Also the following code produces duplicate strings, compared to when
auto-composition-mode is off.

(let ()
  (setq bidi-display-reordering t)
  (insert "\n\n")
  (insert "كمنت")
  (insert "ببببببببببببببببببب"))


[-- Attachment #2: screenshot --]
[-- Type: image/png, Size: 992 bytes --]

[-- Attachment #3: Type: text/plain, Size: 12 bytes --]


-- 
Thamer

[-- Attachment #4: Type: text/plain, Size: 140 bytes --]

_______________________________________________
emacs-bidi mailing list
emacs-bidi@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-bidi

Re: Re: Arabic support

[Kenichi Handa]

In article <87aananet1.fsf@zemblan.newkuwait.org>, Thamer Mahmoud <thamer.mahmoud@gmail.com> writes:

> However, long Arabic strings still have unshaped middle parts and bad
> margin.  See the attached screenshot which is the output of M-30-<BAA>
> in an empty buffer.

Ah, I found what is wrong.  In "struct glyph", we now have
only 4 bits to store indices into an LGSTRING.

    struct
    {
      /* Flag to tell if the composition is automatic or not.  */
      unsigned automatic : 1;
      /* ID of the composition.  */
      unsigned id    : 23;
      /* Start and end indices of glyphs of the composition.  */
      unsigned from : 4;
      unsigned to : 4;
    } cmp;

So, we could handle at most 16 glyphs in one composition.
I've just installed a fix to remove that restriction
(theoretically we still have a restriction of at most
0x7FFFFFFF glyphs in one composition).

---
Kenichi Handa
handa@m17n.org