Merging the bidi branch

Merging the bidi branch

[Eli Zaretskii]

I'm getting ready for the merge of the bidi branch with the trunk.  It
built successfully on MS-Windows and passed some simple testing in a
GUI session.  (Yes, I promised this only in 2 weeks time; I lied.)  So
if everything goes according to plan, I will merge before noon
tomorrow (GMT+3).

Any last-minute reasons why I shouldn't do it just yet?  Speak now or
forever hold your peace.

Re: Merging the bidi branch

[Stefan Monnier]

> I'm getting ready for the merge of the bidi branch with the trunk.  It
> built successfully on MS-Windows and passed some simple testing in a
> GUI session.  (Yes, I promised this only in 2 weeks time; I lied.)  So
> if everything goes according to plan, I will merge before noon
> tomorrow (GMT+3).

Yay!


        Stefan

Re: Merging the bidi branch

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Mon, 29 Mar 2010 10:41:33 -0400
> 
> > I'm getting ready for the merge of the bidi branch with the trunk.  It
> > built successfully on MS-Windows and passed some simple testing in a
> > GUI session.  (Yes, I promised this only in 2 weeks time; I lied.)  So
> > if everything goes according to plan, I will merge before noon
> > tomorrow (GMT+3).
> 
> Yay!

It's done.

However, given the excitement that this feature generates, I would
like to lower everybody's expectations somewhat, to avoid
disappointment and frustration.

The support for bidirectional editing, as it was merged into the
current trunk is still very much experimental.  I do not recommend
turning it on for any purpose but hacking or debugging it.  The chance
of losing your work or crashing Emacs while working in a
bidi-reordered buffer is quite high.  Basic Emacs features has not yet
been made to work reliably in such buffers.  Vertical cursor motion is
a prominent example: it mostly works, but is known to infloop or crash
under some circumstances (because the underlying primitives use
`current_column' and `move_to_column' which assume unidirectional
display).  Use C-f, C-b, M-<, and M-> instead: they do work.

Faces, region highlighting, and invisible text work, but `display'
properties with string values do not.  Display of images was not yet
tested, and in any case, images will probably not be reordered
correctly wrt surrounding bidirectional text, with the current code.
Composed characters will probably cause messed up display (try "C-h H"
and see for yourself).

Etc., etc.

YOU HAVE BEEN WARNED!

That said, if you are still un-intimidated, read the entry in NEWS and
the section in the manual to which it points, and start using and
hacking this new feature.

I need help in developing this further.  The amount of work required
to bring this to the level where it can be released is enormous.  The
good news are that a large portion of the job does not need any
knowledge of bidirectional scripts or UAX#9.

Here are a few tasks that are quite high on my todo and where I could
use help:

 . Display of right-to-left paragraphs.  This currently works
   correctly only on a TTY.  I need help from experts on GUI back-ends
   (X, MS-Windows, and NS) to make it work in GUI sessions.  Each
   glyph_row structure that needs to be displayed starting at the
   right margin has a special flag reversed_p turned on, and its
   glyphs are already in the reversed order.  The back-end needs to
   implement two things:

     o when this flag is on, draw the glyphs flushed all the way to
       the right margin of the window, and

     o if the row width is less than the window's, extend the default
       face to the left margin of the window

 . The bidi reordering code in src/bidi.c uses a table of
   bidirectional properties of characters.  The current code
   constructs this table from static data that was gleaned from the
   Unicode database, but needs to be manually fixed each time Unicode
   database changes.  This needs to be rewritten to use data in
   uni-bidi.el.  The challenge here is to use a memory-efficient data
   structure and (if the data turns out to be significant) decide when
   to load that data.

 . Similarly for the bidi-mirrored properties of characters: we need
   to use data from the Unicode database through uni-*.el files.  The
   current implementation uses a small table that only supports ASCII
   characters, which is clearly insufficient.

 . Turn on features needed for shaping bidirectional scripts.  This is
   needed for Arabic ligatures, at the very least.  People who are
   familiar with libotf and Uniscribe shaping engines used by Emacs
   are welcome to work on this.  I myself have exactly zero knowledge
   about these issues.  There's some potential difficulty here: these
   shaping engines normally reorder characters for display themselves;
   we need to find a way of getting them to perform the shaping
   without reordering.

 . Display of composite characters.  In addition to making it
   bidi-aware, we need to figure out how to display characters with
   diacriticals, because the current bidi reordering code reorders
   them as well, placing the diacritical before the character.