bug#5977: 24.0.50; Lao HELLO is incorrectly displayed

bug#5977: 24.0.50; Lao HELLO is incorrectly displayed

[Peter Dyballa]

[-- Attachment #1: Type: text/plain, Size: 53 bytes --]

Hello!

The Lao greetings are incorrectly rendered:


[-- Attachment #2: GNU Emacs Hello Lao.png --]
[-- Type: image/png, Size: 2761 bytes --]

[-- Attachment #3: Type: text/plain, Size: 1826 bytes --]




The font used is:

     xft:-unknown-DejaVu Sans Mono-normal-normal-normal-*-14-*-*-*-m-0- 
iso10646-1 (#x468)

Similar effects happen with:

     xft:-unknown-Code2000-normal-normal-normal-*-17-*-*-*-*-0- 
iso10646-1 (#xA30)


When I go forward or backward with the cursor keys in the greetings  
(also in the header) the direction of cursor movement is reversed at  
two or three spots. When I try to select them to be able to copy them  
they disappear.


In GNU Emacs 24.0.50.1 (powerpc-apple-darwin9.8.0, X toolkit, Xaw3d  
scroll bars)
  of 2010-04-17 on Latsche.local
Windowing system distributor `The X.Org Foundation', version  
11.0.10800000
configured using `configure  '--without-sound' '--without-dbus' '-- 
without-pop' '--without-gconf' '--with-x-toolkit=athena' '--enable- 
locallisppath=/Library/Application Support/Emacs/calendar23:/Library/ 
Application Support/Emacs' 'CFLAGS=-g -H -Wno-pointer-sign -pipe -fPIC  
-fno-common -mcpu=7450 -mtune=7450 -faltivec -fast' 'CPPFLAGS='  
'LDFLAGS=' 'CC=gcc-4.2' 'CPP=cpp-4.2''

Important settings:
   value of $LC_ALL: nil
   value of $LC_COLLATE: nil
   value of $LC_CTYPE: de_DE.UTF-8
   value of $LC_MESSAGES: nil
   value of $LC_MONETARY: nil
   value of $LC_NUMERIC: nil
   value of $LC_TIME: nil
   value of $LANG: de_DE.UTF-8
   value of $XMODIFIERS: nil
   locale-coding-system: utf-8-unix
   default enable-multibyte-characters: t

Major mode: Fundamental

Minor modes in effect:
   tooltip-mode: t
   mouse-wheel-mode: t
   tool-bar-mode: t
   menu-bar-mode: t
   file-name-shadow-mode: t
   global-font-lock-mode: t
   font-lock-mode: t
   blink-cursor-mode: t
   auto-encryption-mode: t
   auto-compression-mode: t
   line-number-mode: t
   transient-mark-mode: t
   view-mode: t


--
Greetings

   Pete

Time is an illusion. Lunchtime, doubly so.

bug#5977: 24.0.50; Lao HELLO is incorrectly displayed

[Jason Rumney]

Peter Dyballa <Peter_Dyballa@Freenet.DE> writes:

> The Lao greetings are incorrectly rendered:

This seems to be the case with all the scripts that use otf to display
since the bidi code was merged.

bug#5977: 24.0.50; Lao HELLO is incorrectly displayed

[Eli Zaretskii]

> From: Peter Dyballa <Peter_Dyballa@Freenet.DE>
> Date: Mon, 19 Apr 2010 22:50:11 +0200
> Cc: 
> 
> The Lao greetings are incorrectly rendered:

Does the problem go away if you type

    M-: (setq bidi-display-reordering nil) RET

?

> When I go forward or backward with the cursor keys in the greetings  
> (also in the header) the direction of cursor movement is reversed at  
> two or three spots.

I don't see cursor movement reversal on MS-Windows.  Can you tell what
characters are those where it happens, e.g., by counting the number of
C-f keystrokes from the beginning of the line?

Anyway, the problem is known: Lao (as well as quite a few other
scripts and display features in Emacs) use character compositions,
and the bidi display does not yet handle composed characters
correctly.  I need Handa-san's help in figuring out how to make
compositions work with bidi display, because I lack the necessary
knowledge of how support for character compositions is designed and
implemented, and the code is not documented enough, at least not for
me, to figure that out on my own.

bug#5977: 24.0.50; Lao HELLO is incorrectly displayed

[Eli Zaretskii]

> From: Jason Rumney <jasonr@gnu.org>
> Date: Tue, 20 Apr 2010 07:15:49 +0800
> Cc: 5977@debbugs.gnu.org
> 
> Peter Dyballa <Peter_Dyballa@Freenet.DE> writes:
> 
> > The Lao greetings are incorrectly rendered:
> 
> This seems to be the case with all the scripts that use otf to display
> since the bidi code was merged.

Are you saying that you don't see similar problems on MS-Windows?
Because I do.

The Lao script does not use any R2L characters (AFAICS), so the only
issue with the bidi code here is character compositions.  Other than
that, the characters are not reordered, so cursor motion should be
strictly left to right, as usual (although jumps are expected, due to
problems with composed characters).  All this code is in terminal
independent parts of the display engine, so it should affect all GUI
ports in the same way.  The only difference between Uniscribe and
libotf should be in how they render un-composed characters, such as
u+0EB5, that the bidi redisplay incorrectly puts on the screen.

Somebody who knows how character compositions work in Emacs, please
help me to DTRT with them in the bidi display!

bug#5977: 24.0.50; Lao HELLO is incorrectly displayed

[Peter Dyballa]

Am 20.04.2010 um 11:06 schrieb Eli Zaretskii:

>> The Lao greetings are incorrectly rendered:
>
> Does the problem go away if you type
>
>    M-: (setq bidi-display-reordering nil) RET

YES! Many displayed strings change their shape afterwards.

>
> ?
>
>> When I go forward or backward with the cursor keys in the greetings
>> (also in the header) the direction of cursor movement is reversed at
>> two or three spots.
>
> I don't see cursor movement reversal on MS-Windows.  Can you tell what
> characters are those where it happens, e.g., by counting the number of
> C-f keystrokes from the beginning of the line?


When I start at the default minor-mode at the left-most character, the  
strings first change their look. First <right> position point on the  
third character. When I now invoke C-u C-x = on this character point  
goes left to the second character. Next <right> changes again the  
shape of the strings. Next <right> warps point to fifth character,  
next <right> back to fourth, next <right> to seventh, then comes  
<SPACE>, then warp to /, <SPACE>, first character of second string.  
Next <right> positions point on second character and one more <right>  
leaps to possibly fourth one, then fifth (neighbour), then jump to  
possibly seventh, eighth, last ninth, before it falls down on next  
line right of the small depressed Ω like character (see my screenshot).

With bidi-display-reordering set to nil no change in direction of  
movement happens. (Cursor also stops at final C-j of line. Next  
<right> does the line-feed and carriage-return.)

--
Greetings

   Pete

Time flies like an error – but fruit flies like a banana!
				- (almost) Groucho Marx

bug#5977: 24.0.50; Lao HELLO is incorrectly displayed

[Eli Zaretskii]

> Cc: 5977@debbugs.gnu.org
> From: Peter Dyballa <Peter_Dyballa@Freenet.DE>
> Date: Tue, 20 Apr 2010 12:28:11 +0200
> 
> When I start at the default minor-mode at the left-most character, the  
> strings first change their look. First <right> position point on the  
> third character. When I now invoke C-u C-x = on this character point  
> goes left to the second character. Next <right> changes again the  
> shape of the strings. Next <right> warps point to fifth character,  
> next <right> back to fourth, next <right> to seventh, then comes  
> <SPACE>, then warp to /, <SPACE>, first character of second string.  
> Next <right> positions point on second character and one more <right>  
> leaps to possibly fourth one, then fifth (neighbour), then jump to  
> possibly seventh, eighth, last ninth, before it falls down on next  
> line right of the small depressed Ω like character (see my screenshot).

Strange, I see none of this on MS-Windows.  Cursor positioning is also
device-independent code, so I'm probably missing something.

bug#5977: 24.0.50; Lao HELLO is incorrectly displayed

[Jason Rumney]

Eli Zaretskii <eliz@gnu.org> writes:

> Are you saying that you don't see similar problems on MS-Windows?
> Because I do.

I haven't tried on Windows.  But on Linux, all the Indic scripts are
displaying incorrectly, some showing on two lines instead of one, and
cursor movement through them makes the display change - probably an
indication that Emacs thinks it is displaying something different than
what is actually displayed at that point.

> The Lao script does not use any R2L characters (AFAICS), so the only
> issue with the bidi code here is character compositions.

Yes, composition might be the common factor here, though I didn't see
any problem with Thai, which also uses composition but not
libotf/uniscribe.  I did not mean that it had anything to do with R2L,
just that the bidi merge was the point in time that this broke.

bug#5977: 24.0.50; Lao HELLO is incorrectly displayed

[Kenichi Handa]

Sorry for the late response on this matter.

In article <83fx2q5w86.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> Anyway, the problem is known: Lao (as well as quite a few other
> scripts and display features in Emacs) use character compositions,
> and the bidi display does not yet handle composed characters
> correctly.  I need Handa-san's help in figuring out how to make
> compositions work with bidi display, because I lack the necessary
> knowledge of how support for character compositions is designed and
> implemented, and the code is not documented enough, at least not for
> me, to figure that out on my own.

I've been using the branch for 23.2.  I've just build the
trunk code on GNU/Linus, and found that all characters
displayed by composition are incorrect.  But, at the moment,
I don't have a time to work on the trunk.

Here's a brief explanation about control flow.

At first, composition_compute_stop_pos is called in
compute_stop_pos and reseat_to_string to record where to
stop for composition handling in this member
     struct composition_it cmp_it;
of struct it.

Next, next_element_from_string and next_element_from_buffer
calls the macro CHAR_COMPOSED_P to check if the next element
should be composed.  CHAR_COMPOSED_P calls
composition_reseat_it which is the function to compose
character(s) and build a LGSTRING (lispy glyph string) that
carries all information about how to display that character
sequence (glyph-ids of a font, relative position, etc).
When a LGSTRING is built, it's cached and the ID of the
cached data is recorded in cmp_it (see above).

If composition_reseat_it successfully built a LGSTRING,
next_element_from_string and next_element_from_buffer call
next_element_from_composition.

next_element_from_composition sets it->what to
IT_COMPOSITION and setups it->cmp_it so that
x_draw_composite_glyph_string_foreground (called in
x_draw_glyph_string) can draw actual composition glyph(s).

---
Kenichi Handa
handa@m17n.org

bug#5977: 24.0.50; Lao HELLO is incorrectly displayed

[Eli Zaretskii]

> Cc: 5977@debbugs.gnu.org
> From: Peter Dyballa <Peter_Dyballa@Freenet.DE>
> Date: Tue, 20 Apr 2010 12:28:11 +0200
> 
> 
> Am 20.04.2010 um 11:06 schrieb Eli Zaretskii:
> 
> >> The Lao greetings are incorrectly rendered:
> >
> > Does the problem go away if you type
> >
> >    M-: (setq bidi-display-reordering nil) RET
> 
> YES! Many displayed strings change their shape afterwards.

I think I fixed this (revno 100013), please see if the problem is
solved for you.

(The lines in HELLO for Arabic and Hebrew still display slightly
incorrectly, but that's expected, since the fixed code only supports
strict L2R scripts.  Doing that for bidirectional scripts is on my
TODO.)