Bidirectional editing in Emacs -- main design decisions

Bidirectional editing in Emacs -- main design decisions

[Eli Zaretskii]

As some of you know, I'm slowly working on adding support for
bidirectional editing in Emacs.  (Before you ask: the code is not
publicly available yet, and won't be until Emacs switches to bzr as
its main VCS.)

While there's a lot of turf to be covered yet, I thought I'd publish
the main design decisions up to this point.  Many of these decisions
were discussed at length years ago on emacs-bidi mailing list, and
since then I also talked them over in private email with a few people.
Other decisions were made recently, as I went about changing the
display engine.

My goal, and the main drive behind these design decisions was to
preserve as much as possible the basic assumptions and design
principles of the current Emacs display engine.  This is not just
opportunism; I firmly believe that any other way would mean a total
redesign and rewrite of the display engine, which is something we want
to avoid.  Personally, if such a redesign would be necessary, I
couldn't have participated in that endeavor, except as advisor.

With that preamble out of my way, here's what I can tell about the
subject at this point:

1. Text storage

   Bidirectional text in Emacs buffers and strings is stored in strict
   logical order (a.k.a. "reading order").  This is how most (if not
   all) other implementations handle bidirectional text.  The
   advantage of this is that file and process I/O is trivial, as well
   as text search.  The disadvantage is that text needs to be
   reordered for display (see below) and also for sending to any other
   visual-order stream, such as a printer or a file in visual-order
   encoding.

2. Support for Unicode Bidirectional Algorithm

   The Unicode Bidirectional Algorithm, described in Annex 9 of the
   Unicode Standard (a.k.a. UAX#9, see http://www.unicode.org/reports/tr9/),
   specifies how to reorder bidirectional text from logical to visual
   order.  Emacs will belong to the so-called "Full Bidirectionality"
   class of applications, which include support for both implicit
   bidirectional reordering and explicit directional embedding codes
   that allow to override the implicit reordering.  This means that
   Emacs supports the entire spectrum of Unicode character properties
   and special codes relevant to bidirectional text.

3. Bidi formatting codes are retained

   At some point in the reordering described by UAX#9, the various
   formatting codes are to be removed from the text, once they've
   performed their role of forcing the order of characters for
   display, because they are not supposed to be visible on display.
   Contrary to this, Emacs does not remove these formatting codes, it
   just behaves as if they are not there. (This behavior is
   acknowledged by UAX#9 under "Retaining Format Codes" clause, so
   Emacs does not break conformance here.)  This is primarily because
   Emacs must preserve the text that was not edited; in particular,
   visiting a file and then saving it to a different file without
   changing anything must produce the same byte stream as the original
   file, even if the formatting codes were part of the original file.
   In addition, being able to show these formatting codes to the user
   is a valuable feature, because the way reordered text looks might
   not be otherwise understood or changed easily.

4. Reordering of text for display

   Reordering for display happens as part of Emacs redisplay.  In a
   nutshell, the current unidirectional redisplay code walks through
   buffer text and considers each character in turn.  After each
   character is processed and translated into a `struct glyph', which
   includes all the information needed for displaying that character,
   the iterator's position is incremented to the next character.

   In the bidi Emacs, this _linear_ iteration through the buffer is
   replaced with a _non-linear_ one, whereby instead of incrementing
   buffer position, a function is called to return the next position
   in the visual order.  Whatever position it returns is processed
   next into a `struct glyph'.  The rest of the code that produces
   "glyph matrices" (data structures used to decide which parts of the
   screen need to be redrawn) is largely ignorant of the
   bidirectionality of the text.  Of course, parts of the display
   engine that manipulate the glyph matrices directly and assume that
   buffer positions increase monotonically with glyph positions need
   to be fixed or rewritten.  But these parts of the display are
   relatively few and localized.  Also, some redisplay optimizations
   need to be disabled when bidirectional text is rendered for
   display.

5. Visual-order information is volatile

   There were lots of discussions several years ago about whether
   Emacs should record in some way the information needed to reorder
   text into visual order of the characters, to reuse it later.  In
   UAX#9 terminology, this information is the "resolved level" of each
   character.  Various features were suggested as a vehicle for this,
   for example, some special text properties (except that text
   properties, unlike resolved levels, cannot overlap).  Lots of
   energy went into discussing how this information would be recorded
   and how it will be reused, e.g. if portion of the text was
   copy-pasted into a different buffer or string.  The complications,
   it turns out, are abound.

   The current design doesn't record this information at all.  It
   simply recomputes it each time a buffer or string need to be
   displayed or sent to a visual-order stream.  The resolved levels
   are computed during reordering, then forgotten.  It turns out that
   bidirectional iteration through buffer text is not much more
   expensive than the current unidirectional one.  The implementation
   of UAX#9 written for Emacs is efficient enough to make any
   long-term caching of resolved levels unnecessary.

6. Reordering of strings from `display' properties

   Strings that are values of `display' text properties and overlay
   properties are reordered individually.  This matters when such
   properties cover adjacent portions of buffer text, back to back.
   For example, PROP1 is associated with buffer positions P1 to P2,
   and PROP2 immediately follows it, being associated with positions
   P2 to P3.  The current design calls for reordering the characters
   of the strings that are the values of PROP1 and PROP2 separately.
   An alternative would be to feed them concatenated into the
   reordering algorithm, in which case the characters coming from
   PROP2 could end up displayed before (to the left) of the characters
   coming from PROP1.  However, this alternative requires a major
   surgery of several parts of the display code.  (Interested readers
   are advised to read the code of set_cursor_from_row in xdisp.c, as
   just one example.)  It's not clear what is TRT to do in this case
   anyway; I'm not aware of any other application that provides
   similar features, so there's nothing I could compare it to.  So I
   decided to go with the easier design.  If the application needs a
   single long string, it can always collapse two or more `display'
   properties into one long one.

   Another, perhaps more serious implication of this design decision
   is that strings from `display' properties are reordered separately
   from the surrounding buffer text.  IOW, production of glyphs from
   reordered buffer text is stopped when a `display' property is
   found, the string that is the property's value is reordered and
   displayed, and then the rest of text is reordered and its glyphs
   produced.  The effect will be visible, e.g., when a `display'
   string is embedded in right-to-left text in otherwise left-to-right
   paragraph text.  Again, I think in the absence of clear "prior
   art", simplicity of design and the amount of changes required in
   the existing display engine win here.

7. Paragraph base direction

   Bidirectional text can be rendered in left-to-right or in
   right-to-left paragraphs.  The former is used for mostly
   left-to-right text, possibly with some embedded right-to-left text.
   The latter is used for text that is mostly or entirely
   right-to-left.  Right-to-left paragraphs are displayed flushed all
   the way to the right margin of the display; this is how users of
   right-to-left scripts expect to see text in their languages.

   UAX#9 specifies how to determine whether this attribute of a
   paragraph, called "base direction", is one or the other, by finding
   the first strong directional character in the paragraph.  However,
   the Unicode Character Database specifies that NL and CR characters
   are paragraph separators, which means each line is a separate
   paragraph, as far as UAX#9 is concerned.  If Emacs would follow
   UAX#9 to the letter, each line could have different base direction,
   which is, of course, intolerable.  We could avoid this nonsense by
   using the "soft newline" or similar features, but I firmly believe
   that Emacs should DTRT with bidirectional text even in the simplest
   modes, including the Fundamental mode, where every newline is hard.

   Fortunately, UAX#9 acknowledges that applications could have other
   ideas about what is a "paragraph".  It calls this ``higher
   protocol''.  So I decided to use such a higher protocol -- namely,
   the Emacs definition of a paragraph, as determined by the
   `paragraph-start' and `paragraph-separate' regexps.  Therefore, the
   first strong directional character after `paragraph-start' or
   `paragraph-separate' determines the paragraph direction, and that
   direction is kept for all the lines of the paragraph, until another
   `paragraph-separate' is found.  (Of course, this means that
   inserting a single character near the beginning of a paragraph
   might affect the display of all the lines in that paragraph, so
   some of the current redisplay optimizations which deal with changes
   to a single line need to be disabled in this case.)

   There is a buffer-specific variable `paragraph-direction' that
   allows to override this dynamic detection of the direction of each
   paragraph, and force a certain base direction on all paragraphs in
   the buffer.  I expect, for example, each major mode for a
   programming language to force the left-to-right paragraph
   direction, because programming languages are written left to right,
   and right-to-left scripts appear in such buffers only in strings
   embedded in the program or in comments.

8. User control of visual order

   UAX#9 does not always produce perfect results on the screen.
   Notable cases where it doesn't are related to characters such as
   `+' and `-' which have more than one role: they can be used in
   mathematical context or in plain-text context; the "correct"
   reordering turns out to be different in each case.

   Again, lots of energy was invested in past discussions how to
   prevent these blunders.  Several clever heuristics are known to
   avoid that.  The problem is that all those heuristics contradict
   UAX#9, which means text that looks OK in Emacs will look different
   (i.e. wrong) in another application.

   I decided it was unjustified to deviate from UAX#9.  Its algorithm
   already provides the solution to this problem: users can always
   control the visual order by inserting special formatting codes at
   strategic places.  These codes are by default not shown in the
   displayed text, but they influence the resolved directionality of
   the surrounding characters, and thus change their visual order.  We
   could (and probably should) have commands in Emacs to control the
   visual order that will work simply by inserting the appropriate
   formatting codes.  For example, a paragraph starting with an Arabic
   letter could nonetheless be rendered as left-to-right paragraph by
   inserting the LRM code before that Arabic character; Emacs could
   have a command called, say, `make-paragraph-left-to-right' that did
   its job simply by inserting LRM at the beginning of the paragraph.

   This design kills two birds: (a) it produces text that is compliant
   with other applications, and will display the same as in Emacs, and
   (b) it avoids the need to invent yet another Emacs infrastructure
   feature to keep information such as paragraph direction outside of
   the text itself.

That is all for now.  If you have comments or questions, you are
welcome to voice them.  However, I reserver the right to respond only
to those I'm interested in and/or have time for. ;-)

Re: Bidirectional editing in Emacs -- main design decisions

[joakim]

Eli Zaretskii <eliz@gnu.org> writes:

> As some of you know, I'm slowly working on adding support for
> bidirectional editing in Emacs.  (Before you ask: the code is not
> publicly available yet, and won't be until Emacs switches to bzr as
> its main VCS.)

Short question if you have time: I'm working (also slowly) on a patch to
embedd gtk widgets in buffers(the xwidget patch). It works mostly the
same as embedding images. From what youre writing below it sounds like
the display of images will work as before, therefore my patch will
apply, hopefully nicely, on top of bidi. Correct?

It will be very nice to see your patch when youre ready to publish!

> While there's a lot of turf to be covered yet, I thought I'd publish
> the main design decisions up to this point.  Many of these decisions
> were discussed at length years ago on emacs-bidi mailing list, and
> since then I also talked them over in private email with a few people.
> Other decisions were made recently, as I went about changing the
> display engine.
>
> My goal, and the main drive behind these design decisions was to
> preserve as much as possible the basic assumptions and design
> principles of the current Emacs display engine.  This is not just
> opportunism; I firmly believe that any other way would mean a total
> redesign and rewrite of the display engine, which is something we want
> to avoid.  Personally, if such a redesign would be necessary, I
> couldn't have participated in that endeavor, except as advisor.
>
> With that preamble out of my way, here's what I can tell about the
> subject at this point:
>
> 1. Text storage
>
>    Bidirectional text in Emacs buffers and strings is stored in strict
>    logical order (a.k.a. "reading order").  This is how most (if not
>    all) other implementations handle bidirectional text.  The
>    advantage of this is that file and process I/O is trivial, as well
>    as text search.  The disadvantage is that text needs to be
>    reordered for display (see below) and also for sending to any other
>    visual-order stream, such as a printer or a file in visual-order
>    encoding.
>
> 2. Support for Unicode Bidirectional Algorithm
>
>    The Unicode Bidirectional Algorithm, described in Annex 9 of the
>    Unicode Standard (a.k.a. UAX#9, see http://www.unicode.org/reports/tr9/),
>    specifies how to reorder bidirectional text from logical to visual
>    order.  Emacs will belong to the so-called "Full Bidirectionality"
>    class of applications, which include support for both implicit
>    bidirectional reordering and explicit directional embedding codes
>    that allow to override the implicit reordering.  This means that
>    Emacs supports the entire spectrum of Unicode character properties
>    and special codes relevant to bidirectional text.
>
> 3. Bidi formatting codes are retained
>
>    At some point in the reordering described by UAX#9, the various
>    formatting codes are to be removed from the text, once they've
>    performed their role of forcing the order of characters for
>    display, because they are not supposed to be visible on display.
>    Contrary to this, Emacs does not remove these formatting codes, it
>    just behaves as if they are not there. (This behavior is
>    acknowledged by UAX#9 under "Retaining Format Codes" clause, so
>    Emacs does not break conformance here.)  This is primarily because
>    Emacs must preserve the text that was not edited; in particular,
>    visiting a file and then saving it to a different file without
>    changing anything must produce the same byte stream as the original
>    file, even if the formatting codes were part of the original file.
>    In addition, being able to show these formatting codes to the user
>    is a valuable feature, because the way reordered text looks might
>    not be otherwise understood or changed easily.
>
> 4. Reordering of text for display
>
>    Reordering for display happens as part of Emacs redisplay.  In a
>    nutshell, the current unidirectional redisplay code walks through
>    buffer text and considers each character in turn.  After each
>    character is processed and translated into a `struct glyph', which
>    includes all the information needed for displaying that character,
>    the iterator's position is incremented to the next character.
>
>    In the bidi Emacs, this _linear_ iteration through the buffer is
>    replaced with a _non-linear_ one, whereby instead of incrementing
>    buffer position, a function is called to return the next position
>    in the visual order.  Whatever position it returns is processed
>    next into a `struct glyph'.  The rest of the code that produces
>    "glyph matrices" (data structures used to decide which parts of the
>    screen need to be redrawn) is largely ignorant of the
>    bidirectionality of the text.  Of course, parts of the display
>    engine that manipulate the glyph matrices directly and assume that
>    buffer positions increase monotonically with glyph positions need
>    to be fixed or rewritten.  But these parts of the display are
>    relatively few and localized.  Also, some redisplay optimizations
>    need to be disabled when bidirectional text is rendered for
>    display.
>
> 5. Visual-order information is volatile
>
>    There were lots of discussions several years ago about whether
>    Emacs should record in some way the information needed to reorder
>    text into visual order of the characters, to reuse it later.  In
>    UAX#9 terminology, this information is the "resolved level" of each
>    character.  Various features were suggested as a vehicle for this,
>    for example, some special text properties (except that text
>    properties, unlike resolved levels, cannot overlap).  Lots of
>    energy went into discussing how this information would be recorded
>    and how it will be reused, e.g. if portion of the text was
>    copy-pasted into a different buffer or string.  The complications,
>    it turns out, are abound.
>
>    The current design doesn't record this information at all.  It
>    simply recomputes it each time a buffer or string need to be
>    displayed or sent to a visual-order stream.  The resolved levels
>    are computed during reordering, then forgotten.  It turns out that
>    bidirectional iteration through buffer text is not much more
>    expensive than the current unidirectional one.  The implementation
>    of UAX#9 written for Emacs is efficient enough to make any
>    long-term caching of resolved levels unnecessary.
>
> 6. Reordering of strings from `display' properties
>
>    Strings that are values of `display' text properties and overlay
>    properties are reordered individually.  This matters when such
>    properties cover adjacent portions of buffer text, back to back.
>    For example, PROP1 is associated with buffer positions P1 to P2,
>    and PROP2 immediately follows it, being associated with positions
>    P2 to P3.  The current design calls for reordering the characters
>    of the strings that are the values of PROP1 and PROP2 separately.
>    An alternative would be to feed them concatenated into the
>    reordering algorithm, in which case the characters coming from
>    PROP2 could end up displayed before (to the left) of the characters
>    coming from PROP1.  However, this alternative requires a major
>    surgery of several parts of the display code.  (Interested readers
>    are advised to read the code of set_cursor_from_row in xdisp.c, as
>    just one example.)  It's not clear what is TRT to do in this case
>    anyway; I'm not aware of any other application that provides
>    similar features, so there's nothing I could compare it to.  So I
>    decided to go with the easier design.  If the application needs a
>    single long string, it can always collapse two or more `display'
>    properties into one long one.
>
>    Another, perhaps more serious implication of this design decision
>    is that strings from `display' properties are reordered separately
>    from the surrounding buffer text.  IOW, production of glyphs from
>    reordered buffer text is stopped when a `display' property is
>    found, the string that is the property's value is reordered and
>    displayed, and then the rest of text is reordered and its glyphs
>    produced.  The effect will be visible, e.g., when a `display'
>    string is embedded in right-to-left text in otherwise left-to-right
>    paragraph text.  Again, I think in the absence of clear "prior
>    art", simplicity of design and the amount of changes required in
>    the existing display engine win here.
>
> 7. Paragraph base direction
>
>    Bidirectional text can be rendered in left-to-right or in
>    right-to-left paragraphs.  The former is used for mostly
>    left-to-right text, possibly with some embedded right-to-left text.
>    The latter is used for text that is mostly or entirely
>    right-to-left.  Right-to-left paragraphs are displayed flushed all
>    the way to the right margin of the display; this is how users of
>    right-to-left scripts expect to see text in their languages.
>
>    UAX#9 specifies how to determine whether this attribute of a
>    paragraph, called "base direction", is one or the other, by finding
>    the first strong directional character in the paragraph.  However,
>    the Unicode Character Database specifies that NL and CR characters
>    are paragraph separators, which means each line is a separate
>    paragraph, as far as UAX#9 is concerned.  If Emacs would follow
>    UAX#9 to the letter, each line could have different base direction,
>    which is, of course, intolerable.  We could avoid this nonsense by
>    using the "soft newline" or similar features, but I firmly believe
>    that Emacs should DTRT with bidirectional text even in the simplest
>    modes, including the Fundamental mode, where every newline is hard.
>
>    Fortunately, UAX#9 acknowledges that applications could have other
>    ideas about what is a "paragraph".  It calls this ``higher
>    protocol''.  So I decided to use such a higher protocol -- namely,
>    the Emacs definition of a paragraph, as determined by the
>    `paragraph-start' and `paragraph-separate' regexps.  Therefore, the
>    first strong directional character after `paragraph-start' or
>    `paragraph-separate' determines the paragraph direction, and that
>    direction is kept for all the lines of the paragraph, until another
>    `paragraph-separate' is found.  (Of course, this means that
>    inserting a single character near the beginning of a paragraph
>    might affect the display of all the lines in that paragraph, so
>    some of the current redisplay optimizations which deal with changes
>    to a single line need to be disabled in this case.)
>
>    There is a buffer-specific variable `paragraph-direction' that
>    allows to override this dynamic detection of the direction of each
>    paragraph, and force a certain base direction on all paragraphs in
>    the buffer.  I expect, for example, each major mode for a
>    programming language to force the left-to-right paragraph
>    direction, because programming languages are written left to right,
>    and right-to-left scripts appear in such buffers only in strings
>    embedded in the program or in comments.
>
> 8. User control of visual order
>
>    UAX#9 does not always produce perfect results on the screen.
>    Notable cases where it doesn't are related to characters such as
>    `+' and `-' which have more than one role: they can be used in
>    mathematical context or in plain-text context; the "correct"
>    reordering turns out to be different in each case.
>
>    Again, lots of energy was invested in past discussions how to
>    prevent these blunders.  Several clever heuristics are known to
>    avoid that.  The problem is that all those heuristics contradict
>    UAX#9, which means text that looks OK in Emacs will look different
>    (i.e. wrong) in another application.
>
>    I decided it was unjustified to deviate from UAX#9.  Its algorithm
>    already provides the solution to this problem: users can always
>    control the visual order by inserting special formatting codes at
>    strategic places.  These codes are by default not shown in the
>    displayed text, but they influence the resolved directionality of
>    the surrounding characters, and thus change their visual order.  We
>    could (and probably should) have commands in Emacs to control the
>    visual order that will work simply by inserting the appropriate
>    formatting codes.  For example, a paragraph starting with an Arabic
>    letter could nonetheless be rendered as left-to-right paragraph by
>    inserting the LRM code before that Arabic character; Emacs could
>    have a command called, say, `make-paragraph-left-to-right' that did
>    its job simply by inserting LRM at the beginning of the paragraph.
>
>    This design kills two birds: (a) it produces text that is compliant
>    with other applications, and will display the same as in Emacs, and
>    (b) it avoids the need to invent yet another Emacs infrastructure
>    feature to keep information such as paragraph direction outside of
>    the text itself.
>
> That is all for now.  If you have comments or questions, you are
> welcome to voice them.  However, I reserver the right to respond only
> to those I'm interested in and/or have time for. ;-)
>
-- 
Joakim Verona

Re: Bidirectional editing in Emacs -- main design decisions

[Eli Zaretskii]

> From: joakim@verona.se
> Cc: emacs-devel@gnu.org, emacs-bidi@gnu.org
> Date: Fri, 09 Oct 2009 23:55:19 +0200
> 
> It works mostly the same as embedding images. From what youre
> writing below it sounds like the display of images will work as
> before, therefore my patch will apply, hopefully nicely, on top of
> bidi. Correct?

Correct.  Images and any other objects will be reordered according to
UAX#9, and as a single entity.  IOW, imagine that instead of the
embedded widget the buffer has a single character U+FFFC (OBJECT
REPLACEMENT CHARACTER).  The reordering code will treat the embedded
widget as it would treat that character.  That means, in particular,
that if the widget is embedded in text written in some right-to-left
script, the text that precedes the widget will be on the right of the
widget, and text that follows it will be on the left.

Re: Bidirectional editing in Emacs -- main design decisions

[Eli Zaretskii]

> Date: Fri, 09 Oct 2009 23:18:00 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 
> 
>    So I decided to use such a higher protocol -- namely,
>    the Emacs definition of a paragraph, as determined by the
>    `paragraph-start' and `paragraph-separate' regexps.

A small, but significant correction to this: these two regexps are
looked for anchored at line beginning.

The reason for this deliberate deviation from the letter of Emacs
definition of a paragraph are complicated, but the upshot is that from
the user point of view, it does not make sense to change paragraph
direction if the paragraph separator does not begin at the beginning
of a line.

As another deviation from the definition of a paragraph, text that
matches `paragraph-separate' is given the same direction as the
preceding paragraph.  (By contrast, Emacs generally does not consider
`paragraph-separate' as part of any paragraph.)

Re: Bidirectional editing in Emacs -- main design decisions

[joakim]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: joakim@verona.se
>> Cc: emacs-devel@gnu.org, emacs-bidi@gnu.org
>> Date: Fri, 09 Oct 2009 23:55:19 +0200
>> 
>> It works mostly the same as embedding images. From what youre
>> writing below it sounds like the display of images will work as
>> before, therefore my patch will apply, hopefully nicely, on top of
>> bidi. Correct?
>
> Correct.  Images and any other objects will be reordered according to
> UAX#9, and as a single entity.  IOW, imagine that instead of the
> embedded widget the buffer has a single character U+FFFC (OBJECT
> REPLACEMENT CHARACTER).  The reordering code will treat the embedded
> widget as it would treat that character.  That means, in particular,
> that if the widget is embedded in text written in some right-to-left
> script, the text that precedes the widget will be on the right of the
> widget, and text that follows it will be on the left.

Presumably I will also need to tell the widget to render its own text in
bidi mode. 

-- 
Joakim Verona

Re: Bidirectional editing in Emacs -- main design decisions

[Eli Zaretskii]

> From: joakim@verona.se
> Cc: emacs-devel@gnu.org, emacs-bidi@gnu.org
> Date: Sat, 10 Oct 2009 00:42:39 +0200
> 
> Presumably I will also need to tell the widget to render its own text in
> bidi mode. 

By itself, or by using Emacs facilities?

Re: Bidirectional editing in Emacs -- main design decisions

[joakim]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: joakim@verona.se
>> Cc: emacs-devel@gnu.org, emacs-bidi@gnu.org
>> Date: Sat, 10 Oct 2009 00:42:39 +0200
>> 
>> Presumably I will also need to tell the widget to render its own text in
>> bidi mode. 
>
> By itself, or by using Emacs facilities?

They are gtk widgets. I havent looked at it closely, but I presume you
tell the gtk widgets which locale to render text in. So, by itself,
yes. So far I have only tested Swedish.



-- 
Joakim Verona

Re: Bidirectional editing in Emacs -- main design decisions

[Eli Zaretskii]

> From: joakim@verona.se
> Cc: emacs-devel@gnu.org, emacs-bidi@gnu.org
> Date: Sat, 10 Oct 2009 09:28:19 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: joakim@verona.se
> >> Cc: emacs-devel@gnu.org, emacs-bidi@gnu.org
> >> Date: Sat, 10 Oct 2009 00:42:39 +0200
> >> 
> >> Presumably I will also need to tell the widget to render its own text in
> >> bidi mode. 
> >
> > By itself, or by using Emacs facilities?
> 
> They are gtk widgets. I havent looked at it closely, but I presume you
> tell the gtk widgets which locale to render text in.

Actually, I'd expect gtk widgets to do this automatically, no matter
in what locale.  The text it renders should supply the hint.

> So, by itself, yes.

You can do that, Emacs won't care.

Re: Bidirectional editing in Emacs -- main design decisions

[Richard Stallman]

    The reason for this deliberate deviation from the letter of Emacs
    definition of a paragraph are complicated, but the upshot is that from
    the user point of view, it does not make sense to change paragraph
    direction if the paragraph separator does not begin at the beginning
    of a line.

The only case when the paragraph separator does not begin at the
beginning of a line is when the left margin is nonzero.

Why should these paragraphs be different from other paragraphs
with regard to direction of text?

    As another deviation from the definition of a paragraph, text that
    matches `paragraph-separate' is given the same direction as the
    preceding paragraph.  (By contrast, Emacs generally does not consider
    `paragraph-separate' as part of any paragraph.)

I don't think that conflicts at all with the normal definition
of paragraphs.  The separator isn't part of the paragraph,
but its reading direction needs to be determined somehow.

Re: Bidirectional editing in Emacs -- main design decisions

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> CC: emacs-devel@gnu.org, emacs-bidi@gnu.org
> Date: Sat, 10 Oct 2009 05:16:58 -0400
> 
>     The reason for this deliberate deviation from the letter of Emacs
>     definition of a paragraph are complicated, but the upshot is that from
>     the user point of view, it does not make sense to change paragraph
>     direction if the paragraph separator does not begin at the beginning
>     of a line.
> 
> The only case when the paragraph separator does not begin at the
> beginning of a line is when the left margin is nonzero.

I'm not sure I understand the situation you are describing.  (The word
"margin" is too overloaded, even if we confine ourselves to Emacs
parlance alone.)  Could you please provide an example of such a
paragraph?  Then I could reason about it.

> Why should these paragraphs be different from other paragraphs
> with regard to direction of text?

They are not "different", they just follow the base direction of the
preceding paragraph.

Re: Bidirectional editing in Emacs -- main design decisions

[Sascha Wilde]

Eli Zaretskii <eliz@gnu.org> wrote:
> 8. User control of visual order
[...]
>    Emacs could
>    have a command called, say, `make-paragraph-left-to-right' that did
>    its job simply by inserting LRM at the beginning of the paragraph.

I would suggest that Emacs should also have a way to visualize the
otherwise invisible text direction marks so that:

- it becomes transparent to the user whether the direction of an
  specific portion of the text is explicit or implicit defined

- the user is provided with a simple way to remove the marks in case he
  wants to.  (By simply deleting them the way he would do with any other
  regular character.)

cheers
sascha
-- 
Sascha Wilde : "GUIs normally make it simple to accomplish simple 
             : actions and impossible to accomplish complex actions."
             : (Doug Gwyn - 22/Jun/91 in comp.unix.wizards)

Re: Bidirectional editing in Emacs -- main design decisions

[Eli Zaretskii]

> From: Sascha Wilde <wilde@sha-bang.de>
> Cc: emacs-devel@gnu.org,  emacs-bidi@gnu.org
> Date: Sat, 10 Oct 2009 15:44:19 +0200
> 
> Eli Zaretskii <eliz@gnu.org> wrote:
> > 8. User control of visual order
> [...]
> >    Emacs could
> >    have a command called, say, `make-paragraph-left-to-right' that did
> >    its job simply by inserting LRM at the beginning of the paragraph.
> 
> I would suggest that Emacs should also have a way to visualize the
> otherwise invisible text direction marks so that:
> 
> - it becomes transparent to the user whether the direction of an
>   specific portion of the text is explicit or implicit defined
> 
> - the user is provided with a simple way to remove the marks in case he
>   wants to.  (By simply deleting them the way he would do with any other
>   regular character.)

Yes.  That's what this part of my longish message was trying to say:

   In addition, being able to show these formatting codes to the user
   is a valuable feature, because the way reordered text looks might
   not be otherwise understood or changed easily.

The "direction marks" you mention are the "formatting codes" I wrote
about.

Re: Bidirectional editing in Emacs -- main design decisions

[Ehud Karni]

On Fri, 09 Oct 2009 23:18:00 Eli Zaretskii wrote:
>
> Here's what I can tell about the subject (bidi display) at this point

In general I agree with your decisions.

> 1. Text storage
>
>    Bidirectional text in Emacs buffers and strings is stored in strict
>    logical order (a.k.a. "reading order").  This is how most (if not
>    all) other implementations handle bidirectional text.  The
>    advantage of this is that file and process I/O is trivial, as well
>    as text search.  [snip]

The search has many problems but this should not influence your bidi
reordering. The changes to various search functions can be done later.

The user ALWAYS search for the visual text s/he sees (S/he never knows
the logical order unless she visits the file literally).

The problems are caused by many reasons:
  1. Different logical inputs, even without formatting characters, can
     result in the same visual output.
     e.g. Logical Hebrew text + a number in LTR reading order, the
     number may be before or after the Hebrew text, but in the visual
     output the number will always be after (to the left of) the text.
     Logical "123 HEBREW 456" appears as "123 456 WERBEH".
  2. Formatting characters are not seen and should not be searched.
  3. The visual appearance of the searched string may be different from
     what it will match.  e.g. The search for logical "HEBREW 3." in
     RTL reading order will appear as ".3 WERBEH" but will match
     also something like logical "HEBREW 3.14159" which its visual
     appearance is "3.14159 WERBEH". This may be what the user wants
     but it may also disturb her because she really wants to find only
     (visual) ".3 WERBEH".
     There is also a technical question, how Emacs will show the found
     string which is not connected as in the "3.14159 WERBEH" above.

As a minimum adjustment, I think the search must ignore the formatting
characters. An option to show (or operate, in search & replace) only on
found matches that are also the same visually is recommended.

> 3. Bidi formatting codes are retained

Agreed, but see my comment on search.

> 7. Paragraph base direction
>
>    There is a buffer-specific variable `paragraph-direction' that
>    allows to override this dynamic detection of the direction of each
>    paragraph, and force a certain base direction on all paragraphs in
>    the buffer.  I expect, for example, each major mode for a
>    programming language to force the left-to-right paragraph
>    direction, because programming languages are written left to right,
>    and right-to-left scripts appear in such buffers only in strings
>    embedded in the program or in comments.

I think a better name is `bidi-paragraphs-direction' or even
`bidi-paragraphs-reading-direction'. Note the `s' in paragraphs,
because it is influence all the paragraphs in the buffer.

There should be a key to toggle this variable. It will very
useful for the minibuffer.

> 8. User control of visual order

Do you intend to support all the explicit formatting characters (LRO is
specially important as it allows to store visual strings as is) or just
the implicit (and more used) LRM and RLM ?

>    This design kills two birds: (a) it produces text that is compliant
>    with other applications, and will display the same as in Emacs, and
>    (b) it avoids the need to invent yet another Emacs infrastructure
>    feature to keep information such as paragraph direction outside of
>    the text itself.

While you can store the LRM and RLM in ISO-8859-8 encoding, there is no
way to store the the other formatting characters.

> That is all for now.  If you have comments or questions, you are
> welcome to voice them.

I found an editor that support the all the formatting characters, YODIT
(http://www.yudit.org/) it is GPLed, may be you can use it.

The W3C recommend not to use explicit formatting characters (i.e.
RLO/LRO/RLE/LRE/PDF) and instead to use markup (see
http://www.w3.org/International/questions/qa-bidi-controls ,
specially the "reasons" section).

Ehud.


--
 Ehud Karni           Tel: +972-3-7966-561  /"\
 Mivtach - Simon      Fax: +972-3-7976-561  \ /  ASCII Ribbon Campaign
 Insurance agencies   (USA) voice mail and   X   Against   HTML   Mail
 http://www.mvs.co.il  FAX:  1-815-5509341  / \
 GnuPG: 98EA398D <http://www.keyserver.net/>    Better Safe Than Sorry

Re: Bidirectional editing in Emacs -- main design decisions

[Jason Rumney]

Eli Zaretskii wrote:
> 4. Reordering of text for display
>   

Does the function font-shape-gstring help with fitting this in?

> 8. User control of visual order
>   
>  I decided it was unjustified to deviate from UAX#9.  Its algorithm
>  already provides the solution to this problem: users can always
>  control the visual order by inserting special formatting codes at
>  strategic places.

Couldn't Emacs by default use the clever heuristics to decide when to 
automatically insert the special formatting codes? It would have to be 
optional and undoable of course, because heuristics are never perfect, 
but it seems to me as a naive non-speaker of RTL languages that to DWIM 
in these edge cases is the right behaviour.

Also you mention several times that the special direction change codes 
are not displayed, but there should be an option to display them IMHO, 
(perhaps part of whitespace.el) as users may need to distinguish between 
explicit direction changes and implicit ones in some circumstances.

Re: Bidirectional editing in Emacs -- main design decisions

[Sascha Wilde]

Eli Zaretskii <eliz@gnu.org> wrote:
[...]
> Yes.  That's what this part of my longish message was trying to say:
>
>    In addition, being able to show these formatting codes to the user
>    is a valuable feature, because the way reordered text looks might
>    not be otherwise understood or changed easily.

Oh, I see.  I should have read your message even more carefully.  Sorry
for the noise.  ;-)

cheers
sascha
-- 
Sascha Wilde : VI is to EMACS as masturbation is to making love:
             : effective and always available but probably not your
             : first choice...

Re: Bidirectional editing in Emacs -- main design decisions

[Eli Zaretskii]

> Date: Sat, 10 Oct 2009 23:13:48 +0800
> From: Jason Rumney <jasonr@gnu.org>
> CC: emacs-devel@gnu.org, emacs-bidi@gnu.org
> 
> Eli Zaretskii wrote:
> > 4. Reordering of text for display
> >   
> 
> Does the function font-shape-gstring help with fitting this in?

I'm sorry to say that I don't understand what it does.  If you can
explain, or give me an educational example to play with, maybe I will
be able to answer your question, my profound ignorance of GUI
rendering notwithstanding.

> > 8. User control of visual order
> >   
> >  I decided it was unjustified to deviate from UAX#9.  Its algorithm
> >  already provides the solution to this problem: users can always
> >  control the visual order by inserting special formatting codes at
> >  strategic places.
> 
> Couldn't Emacs by default use the clever heuristics to decide when to 
> automatically insert the special formatting codes? It would have to be 
> optional and undoable of course, because heuristics are never perfect, 
> but it seems to me as a naive non-speaker of RTL languages that to DWIM 
> in these edge cases is the right behaviour.

I agree: if it's possible to DWIM automatically, we should.  But you
are talking about higher levels than where I am right now.  For my
purposes, it was good enough to decide that any such clever heuristics
will eventually just add or remove formatting codes, and that no other
infrastructure features are needed to support this.

> Also you mention several times that the special direction change codes 
> are not displayed, but there should be an option to display them IMHO, 
> (perhaps part of whitespace.el) as users may need to distinguish between 
> explicit direction changes and implicit ones in some circumstances.

Yes, that's a very important feature, of course.  Which is why I
tried (but obviously failed, since you are the second person asking
the same) to tell that it will and must be supported:

   In addition, being able to show these formatting codes to the user
   is a valuable feature, because the way reordered text looks might
   not be otherwise understood or changed easily.

Re: Bidirectional editing in Emacs -- main design decisions

[Jason Rumney]

Eli Zaretskii wrote:
>> Does the function font-shape-gstring help with fitting this in?
>>     
>
> I'm sorry to say that I don't understand what it does.  If you can
> explain, or give me an educational example to play with, maybe I will
> be able to answer your question, my profound ignorance of GUI
> rendering notwithstanding.
>   

On second thoughts, it probably doesn't help for two reasons: 
font-shape-gstring uses the font backend to do glyph reordering, which 
would give inconsistent results on different platforms; and I just 
realised that to work on terminals, the font backend is the wrong place 
to be doing bidi reordering.

> Which is why I tried

Apologies for skimming over parts of your message.

Re: Bidirectional editing in Emacs -- main design decisions

[Eli Zaretskii]

> Date: Sat, 10 Oct 2009 16:57:59 +0200
> From: "Ehud Karni" <ehud@unix.mvs.co.il>
> Cc: emacs-bidi@gnu.org, emacs-devel@gnu.org
> 
> On Fri, 09 Oct 2009 23:18:00 Eli Zaretskii wrote:
> >
> > Here's what I can tell about the subject (bidi display) at this point
> 
> In general I agree with your decisions.

Well, you brought up many of them (thanks!), so it isn't surprising ;-)

> The search has many problems but this should not influence your bidi
> reordering. The changes to various search functions can be done later.

Agreed.

> The user ALWAYS search for the visual text s/he sees (S/he never knows
> the logical order unless she visits the file literally).

She will look for visual text, but she will type the text she looks
for in the logical (reading) order, not in the visual order, where
characters are reversed and/or reshuffled.

> The problems are caused by many reasons:
>   1. Different logical inputs, even without formatting characters, can
>      result in the same visual output.
>      e.g. Logical Hebrew text + a number in LTR reading order, the
>      number may be before or after the Hebrew text, but in the visual
>      output the number will always be after (to the left of) the text.
>      Logical "123 HEBREW 456" appears as "123 456 WERBEH".
>   2. Formatting characters are not seen and should not be searched.
>   3. The visual appearance of the searched string may be different from
>      what it will match.  e.g. The search for logical "HEBREW 3." in
>      RTL reading order will appear as ".3 WERBEH" but will match
>      also something like logical "HEBREW 3.14159" which its visual
>      appearance is "3.14159 WERBEH". This may be what the user wants
>      but it may also disturb her because she really wants to find only
>      (visual) ".3 WERBEH".

All of these are valid and important considerations, and the search
commands and primitives will have to deal with them, of course.
There's also the issue of ``final'' letters in Hebrew and much more
complex similar issues in Arabic, etc.  I hope enough
application-level Emacs programmers will come aboard and handle all
this, because otherwise these scripts will never be supported well
enough in Emacs.

However, taking care of this is still quite far in the future.  My
main difficulty in making these decisions was to convince myself that,
while none of these problems are trivial to solve and their solutions
are not even known yet in detail, at least not to me, they are all
_solvable_in_principle_ using just the logical-order text and the
reordering engine (which was designed to allow it to be used by code
other than just the redisplay iterator, so that it's easy to write a
Lisp primitive that takes a logical-order string and returns its
visual-order variant).  Comments that question or contradict this
conclusion are what I'm seeking now, because changing these decisions
further down the road may be very difficult, to say nothing of the
wasted effort.

>      There is also a technical question, how Emacs will show the found
>      string which is not connected as in the "3.14159 WERBEH" above.

I didn't yet adapt support for faces to bidi display.  However, my
plan is to make it so that each character produced by the bidi
iterator gets the correct face, like it does today, and faces are (and
will be in the future) set in the logical order of buffer positions.
So, in your example, the characters underlined below will have the
`isearch' face:

                            3.14159 WERBEH
                            --      ------

(you were saying that the search string is ".3 WERBEH").  Yes, this
shows as disconnected.  But other GUI applications do it that way, so
I think the user will expect this behavior.

> As a minimum adjustment, I think the search must ignore the formatting
> characters.

Yes, of course.  At least by default, with an option to not ignore
them.

> Do you intend to support all the explicit formatting characters (LRO is
> specially important as it allows to store visual strings as is) or just
> the implicit (and more used) LRM and RLM ?

All of them.  They are already supported in the code that I'm using
now.  Like I said, Emacs will support the full set of features
described by UAX#9.

> >    This design kills two birds: (a) it produces text that is compliant
> >    with other applications, and will display the same as in Emacs, and
> >    (b) it avoids the need to invent yet another Emacs infrastructure
> >    feature to keep information such as paragraph direction outside of
> >    the text itself.
> 
> While you can store the LRM and RLM in ISO-8859-8 encoding, there is no
> way to store the the other formatting characters.

UAX#9 recommends to use LRM and RLM, in preference to the other codes,
for this very reason.  Users who will want to use the other codes (in
the rare cases where they are necessary), will have to encode text in
UTF-8.  I don't see this as a serious problem, though: unlike several
years ago, when this issue was discussed at length on emacs-bidi, the
number of applications supporting UTF-8 is very large today.  Heck,
even Notepad groks it nowadays!

> I found an editor that support the all the formatting characters, YODIT
> (http://www.yudit.org/) it is GPLed, may be you can use it.

Thanks, I had it installed already.

> The W3C recommend not to use explicit formatting characters (i.e.
> RLO/LRO/RLE/LRE/PDF) and instead to use markup (see
> http://www.w3.org/International/questions/qa-bidi-controls ,
> specially the "reasons" section).

Yes, I know.  The obsession of W3C with markup is well known ;-)  But
Emacs is first and foremost a _text_editor_, so it doesn't make sense
to me to force users to use markup just to be able to read or write
bidirectional text.  I also believe that converting text that uses
Unicode formatting codes into markup is not such a hard job, and
someone will surely come up soon enough with an Emacs function to do
that.

Re: Bidirectional editing in Emacs -- main design decisions

[James Cloos]

>>>>> "Eli" == Eli Zaretskii <eliz@gnu.org> writes:

Eli> I'm slowly working on adding support for bidirectional editing in Emacs.

Thanks for posting that.  It is a great summary of the concerns and
needs of an editor when dealing with bidi test.

To be fair, I should point out before continuing that I do not read any
rtl scripts.  My interests deal with fonts and typography and at least
seeing bidi email in its correct visual order, if only to try to learn
some of it.

Eli> 1. Text storage
Eli> 2. Support for Unicode Bidirectional Algorithm
Eli> 3. Bidi formatting codes are retained
Eli> 4. Reordering of text for display
Eli> 5. Visual-order information is volatile
Eli> 6. Reordering of strings from `display' properties
Eli> 7. Paragraph base direction
Eli> 8. User control of visual order

Of those points, all but #6 are no brainers; your choices are exactly
what an editor must do.

Point six is an interesting problem; I'm also unaware of any prior
art.  I suspect that in the long term it would be best to note the
start and end directionality of such chunks of text and set them
chunk-by-chunk in a manner similar to how glyphs are set in the
absence of such properties.  But in the short term I agree with
the choice you outlined.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: Bidirectional editing in Emacs -- main design decisions

[Eli Zaretskii]

> From: James Cloos <cloos@jhcloos.com>
> Cc: emacs-devel@gnu.org,  emacs-bidi@gnu.org
> 
> Thanks for posting that.  It is a great summary of the concerns and
> needs of an editor when dealing with bidi test.

Thanks, but I think it's just the beginning.  There are lots of other
issues to deal with; see, for example, the aspects of search described
by Ehud Karni in this thread.

The hard problem in making these decisions was to become convinced
that all those other issues are reasonably solvable based on these
basic features, without actually solving any of them.

> Of those points, all but #6 are no brainers; your choices are exactly
> what an editor must do.

Thanks for confirming that.

> Point six is an interesting problem; I'm also unaware of any prior
> art.  I suspect that in the long term it would be best to note the
> start and end directionality of such chunks of text and set them
> chunk-by-chunk in a manner similar to how glyphs are set in the
> absence of such properties.

I think this is impossible in general, because once text is reordered,
the information needed to plug in additional chunks (the resolved
level of each character) is lost.

Note that it is fairly simple to reorder the text of `display' strings
together with the surrounding text -- you just need to feed the
characters together into the reordering engine.  The problem is
elsewhere -- in the code that uses the produced glyphs.

> But in the short term I agree with the choice you outlined.

The future will tell if it was the right decision.  Maybe a useful
first step to examining its validity would be to prepare a fairly
complete list of Emacs applications that currently use the `display'
text properties and overlay properties.  Given such a list, one could
think of their applicability to bidirectional editing, and how the
strings should be displayed in each context to do what the users
expect.

Re: Bidirectional editing in Emacs -- main design decisions

[Richard Stallman]

    > The only case when the paragraph separator does not begin at the
    > beginning of a line is when the left margin is nonzero.

    I'm not sure I understand the situation you are describing.  (The word
    "margin" is too overloaded, even if we confine ourselves to Emacs
    parlance alone.)  Could you please provide an example of such a
    paragraph?  Then I could reason about it.

I can't give you an example, but move-to-left-margin shows
what it means to be at the left margin.  It is a matter of
matching the paragraph regexps after the right amount of whitespace
as specified by the value of `left-margin'.

When `left-margin' is nonzero, a line which fails to start with that much
whitespace also starts a paragraph.

    > Why should these paragraphs be different from other paragraphs
    > with regard to direction of text?

    They are not "different", they just follow the base direction of the
    preceding paragraph.

To be fully correct, it ought to detect paragraphs correctly when
`left-margin' is nonzero.  I think that won't be hard to do.

Re: Bidirectional editing in Emacs -- main design decisions

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> CC: emacs-devel@gnu.org, emacs-bidi@gnu.org
> Date: Sun, 11 Oct 2009 04:41:22 -0400
> 
> move-to-left-margin shows what it means to be at the left margin.
> It is a matter of matching the paragraph regexps after the right
> amount of whitespace as specified by the value of `left-margin'.

Would it be sufficient to account for any arbitrary amount of
horizontal whitespace between the beginning of the line and the
paragraph regexps?  If so, that is an almost trivial modification of
the code I already have.

My problem was with potentially more complicated situations, since
paragraph regexps may in principle be anything.

I also have issues with a paragraph that is separated from the
previous one by just the amount of indentation, like this:

  aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
  aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
          bbbbbbbbbbbbbbbbbbbbbbbbbbbb
          bbbbbbbbbbbbbbbbbbbbbbbbbbbb

Are you suggesting that Emacs should recompute the paragraph direction
of the two lines with b's, and the result could be a different base
direction from that used by the two preceding lines with a's?  I think
that such direction changes will annoy users of bidirectional scripts.
What are the use-cases where such paragraphs are useful?

> When `left-margin' is nonzero, a line which fails to start with that much
> whitespace also starts a paragraph.

You mean, a line which starts with more indentation, or a line that
starts with less?  Like this:

      aaaaaaaaaaaaaaaaaaaaaa
      aaaaaaaaaaaaaaaaaaaaaa
  xxxxxxxxxxxxxxxxxxxxxxxxxx
      yyyyyyyyyyyyyyyyyyyyyy

Should the "xxx" line be considered a new paragraph?

> To be fully correct, it ought to detect paragraphs correctly when
> `left-margin' is nonzero.  I think that won't be hard to do.

Maybe it will be not hard -- once I make move-to-column,
current-column, and the rest of indent.c work with bidirectional
text.  Right now, it's badly broken, because it assumes buffer
positions increase linearly with screen positions.  Even vertical
cursor motion and C-e does not work correctly, because of that.

Re: Re: Bidirectional editing in Emacs -- main design decisions

[Eli Zaretskii]

> Date: Sun, 11 Oct 2009 22:12:54 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-bidi@gnu.org, emacs-devel@gnu.org
> 
> Maybe it will be not hard -- once I make move-to-column,
> current-column, and the rest of indent.c work with bidirectional
> text.  Right now, it's badly broken

Broken for a buffer with bidirectional text, I should have said.  It
works okay with unidirectional left-to-right text, of course.

Re: Bidirectional editing in Emacs -- main design decisions

[Richard Stallman]

    Would it be sufficient to account for any arbitrary amount of
    horizontal whitespace between the beginning of the line and the
    paragraph regexps?

No, that's not correct.  You need to skip whitespace whose width is
the value of `left-margin' and then match the regexp.
(More precisely, you need to skip the amount of space
specified by the value that the function `current-left-margin' would return.)

Looking at the code, I think I was mistaken in what I said about "less
than `left-margin' indentation starts a paragraph".  I think that if
the line doesn't have `left-margin' worth of indentation, then the
paragaph regexps match at the end of the indentation.

Look at the code of `forward-paragraph' to see the paragraph criteria
in full detail.  It is very important to support the full set of
features that Emacs offers for controlling paragraphs.

At least, it is important to support the full set when this is
released.  I won't say it has to be the very next job you work on.

Re: Bidirectional editing in Emacs -- main design decisions

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> CC: emacs-devel@gnu.org, emacs-bidi@gnu.org
> Date: Mon, 12 Oct 2009 06:11:42 -0400
> 
> It is very important to support the full set of features that Emacs
> offers for controlling paragraphs.

OK, I will add this to my TODO.

Thanks.

Re: Bidirectional editing in Emacs -- main design decisions

[Eli Zaretskii]

> Date: Fri, 09 Oct 2009 23:18:00 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 
> 
> While there's a lot of turf to be covered yet, I thought I'd publish
> the main design decisions up to this point.

I don't know if someone follows these design decisions, but in case
people are interested, here's an update:

> 6. Reordering of strings from `display' properties
>    ...
>    Another, perhaps more serious implication of this design decision
>    is that strings from `display' properties are reordered separately
>    from the surrounding buffer text.  IOW, production of glyphs from
>    reordered buffer text is stopped when a `display' property is
>    found, the string that is the property's value is reordered and
>    displayed, and then the rest of text is reordered and its glyphs
>    produced.

After careful thought, I changed my mind about this part.  Whenever
the redisplay iterator finds text that is covered by a `display' text
property or by an overlay with a `display', `before-string', or
`after-string' property, it will not stop reordering.  Instead, it
will treat the entire run of text covered by the text property or
overlay as a single atomic entity, and will reorder it as if it were a
single special character whose name in Unicode is OBJECT REPLACEMENT
CHARACTER (u+FFFC).  This character's bidirectional category (Other
Neutral) and other properties are designed so that it can stand for
display features such as embedded images, and in particular it is
reordered as appropriate for such embedded objects.

This will reorder images and display strings in the same way wrt
surrounding text, which I think is reasonable.  It is also in line
with the pre-Emacs 24 unidirectional display engine, which skipped the
text covered by such properties in one go, after displaying the image
or a display string specified by the property.

I hope this is the last "main design decision" regarding the basic
bidirectional display.  Barring any unexpected events, I will soon
begin implementing reordering of display strings and images, and what
will be left is "just" debugging.

What will remain in the design department after that is bidirectional
display in buffers that are not plain text: program source files
(where bidirectional scripts can be found in strings and comments) and
markup languages such as HTML and XML.

Re: Bidirectional editing in Emacs -- main design decisions

[Stefan Monnier]

> After careful thought, I changed my mind about this part.  Whenever
> the redisplay iterator finds text that is covered by a `display' text
> property or by an overlay with a `display', `before-string', or
> `after-string' property, it will not stop reordering.  Instead, it
> will treat the entire run of text covered by the text property or
> overlay as a single atomic entity, and will reorder it as if it were a
> single special character whose name in Unicode is OBJECT REPLACEMENT
> CHARACTER (u+FFFC).  This character's bidirectional category (Other
> Neutral) and other properties are designed so that it can stand for
> display features such as embedded images, and in particular it is
> reordered as appropriate for such embedded objects.

> This will reorder images and display strings in the same way wrt
> surrounding text, which I think is reasonable.  It is also in line
> with the pre-Emacs 24 unidirectional display engine, which skipped the
> text covered by such properties in one go, after displaying the image
> or a display string specified by the property.

It sounds very reasonable, but at the same time I don't understand in
which way it differs from your earlier opinion that it should "stop
reordering" (whose meaning is very unclear to me).


        Stefan

Re: Bidirectional editing in Emacs -- main design decisions

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org,  emacs-bidi@gnu.org
> Date: Tue, 19 Apr 2011 10:11:33 -0300
> 
> It sounds very reasonable, but at the same time I don't understand in
> which way it differs from your earlier opinion that it should "stop
> reordering" (whose meaning is very unclear to me).

Well, that's my fault: "stop reordering" is simply misleading.

What I meant by that was that the reordering will completely process
all the text before (in the logical order) the part covered by a
display property, then process the display spec, and only then reorder
the text after the property.

So this buffer text

  ABCDE display XYZ

will be reordered as

  EDCBA display ZYX

instead of more sensible

  ZYX display EDCBA

under the new plan.

Re: Bidirectional editing in Emacs -- main design decisions

[Stefan Monnier]

> Well, that's my fault: "stop reordering" is simply misleading.

> What I meant by that was that the reordering will completely process
> all the text before (in the logical order) the part covered by a
> display property, then process the display spec, and only then reorder
> the text after the property.

> So this buffer text

>   ABCDE display XYZ

> will be reordered as

>   EDCBA display ZYX

> instead of more sensible

>   ZYX display EDCBA

> under the new plan.

Thanks.  That makes sense,


        Stefan

Re: Bidirectional editing in Emacs -- main design decisions

[Mohsen BANAN]

First let me preface that I am new to this list
(through gmane) and that I am not on top of the
subject matter at hand.

If I am understaning it right, I think your new
decision is right also from a different
perspective.

In that it then becomes consistent with how bidi
works in mozilla/firefox.

Being consistent with the browser is important in
practice. Let me cite an example.

I have been using build GNU Emacs 24.0.50.1 for a
while to exchange email (Gnus) in Farsi and mixed
Farsi/Latin with family and friends in Iran who
use browser based MUAs.

My mixed language text generated with emacs
displays inconsistently in firefox vs emacs24.

For example when I write:

اسم کوچک من محسن Mohsen و اسم خانوادگى من بنان
Banan است.


My first name is Mohsen محسن and my last name is
Banan بنان.

And I think the design decision that you mentioned
will make them consistent.

Thanks for the good work.

Best,

...Mohsen

...محسن



>>>>> On Tue, 19 Apr 2011 19:02:04 +0300, Eli Zaretskii <eliz@gnu.org> said:

  >> From: Stefan Monnier <monnier@iro.umontreal.ca>
  >> Cc: emacs-devel@gnu.org,  emacs-bidi@gnu.org
  >> Date: Tue, 19 Apr 2011 10:11:33 -0300
  >> 
  >> It sounds very reasonable, but at the same time I don't understand in
  >> which way it differs from your earlier opinion that it should "stop
  >> reordering" (whose meaning is very unclear to me).

  Eli> Well, that's my fault: "stop reordering" is simply misleading.

  Eli> What I meant by that was that the reordering will completely process
  Eli> all the text before (in the logical order) the part covered by a
  Eli> display property, then process the display spec, and only then reorder
  Eli> the text after the property.

  Eli> So this buffer text

  Eli>   ABCDE display XYZ

  Eli> will be reordered as

  Eli>   EDCBA display ZYX

  Eli> instead of more sensible

  Eli>   ZYX display EDCBA

  Eli> under the new plan.

Re: Bidirectional editing in Emacs -- main design decisions

[Eli Zaretskii]

> From: Mohsen BANAN <list-general@mohsen.1.banan.byname.net>
> Date: Mon, 25 Apr 2011 10:31:57 -0700
> Cc: emacs-devel@gnu.org
> 
> My mixed language text generated with emacs
> displays inconsistently in firefox vs emacs24.
> 
> For example when I write:
> 
> اسم کوچک من محسن Mohsen و اسم خانوادگى من بنان
> Banan است.
> 
> 
> My first name is Mohsen محسن and my last name is
> Banan بنان.

Did you (setq bidi-display-reordering t) in the buffer where this is
displayed?  Without that, the bidirectional reordering does not
happen.  (Some day, this will be on by default, but not yet.)


_______________________________________________
emacs-bidi mailing list
emacs-bidi@gnu.org
https://lists.gnu.org/mailman/listinfo/emacs-bidi

Re: Bidirectional editing in Emacs -- main design decisions

[Mohsen BANAN]

>>>>> On Mon, 25 Apr 2011 20:58:26 +0300, Eli Zaretskii <eliz@gnu.org> said:

  >> From: Mohsen BANAN <list-general@mohsen.1.banan.byname.net>
  >> Date: Mon, 25 Apr 2011 10:31:57 -0700
  >> Cc: emacs-devel@gnu.org
  >> 
  >> My mixed language text generated with emacs
  >> displays inconsistently in firefox vs emacs24.
  >> 
  >> For example when I write:
  >> 
  >> اسم کوچک من محسن Mohsen و اسم خانوادگى من بنان
  >> Banan است.
  >> 
  >> 
  >> My first name is Mohsen محسن and my last name is
  >> Banan بنان.

  Eli> Did you (setq bidi-display-reordering t) in the buffer where this is
  Eli> displayed?  Without that, the bidirectional reordering does not
  Eli> happen.  (Some day, this will be on by default, but not yet.)

Of course Eli! (setq bidi-display-reordering t) is
there when I compose and when I read.

The problem is not with display of a pure Farsi
line or a pure Latin line or a mixed Latin+Farsi
line.

The problem is with Fasri+Latin line. 
In that it displays one way in emacs and a 
differnt way in firefox.

Your citation above of my email has somehow
reproduced the problem.

So, right here we have it captured.

Look at the display of my original message 
below:

اسم کوچک من محسن Mohsen و اسم خانوادگى من بنان
Banan است.

Now look at the display of citation line above
starting on the left with ' >>'  after the 
  >> For example when I write:

You see how the Farsi piece (word sequence -- not
character sequence) is flipped around "Mohsen".

For example look at the dispaly ordering of 
" Mohsen محسن " in the original vs citation.

In the browser my original text appears as it does
in the citation.

This email in and of itself with the citation some
how has produced the same display inconsistency
that I mentioned between emacs24 and firefox.

The problem is with a line starting in Farsi which
includes Latin.

And it displays just fine under emacs24 but word
sequences are flipped when the browser renders
that same byte sequence.

All of this may well be not related to the design
decision that you mentioned but the inconsistency
with the browser rendering is real.

...Mohsen

Re: Bidirectional editing in Emacs -- main design decisions

[Eli Zaretskii]

> From: Mohsen BANAN <list-general@mohsen.1.banan.byname.net>
> Cc: Mohsen BANAN <list-general@mohsen.1.banan.byname.net>,  emacs-bidi@gnu.org,  emacs-devel@gnu.org
> Date: Mon, 25 Apr 2011 11:44:06 -0700
> 
> The problem is not with display of a pure Farsi
> line or a pure Latin line or a mixed Latin+Farsi
> line.
> 
> The problem is with Fasri+Latin line. 
> In that it displays one way in emacs and a 
> differnt way in firefox.
> 
> Your citation above of my email has somehow
> reproduced the problem.
> 
> So, right here we have it captured.
> 
> Look at the display of my original message 
> below:
> 
> اسم کوچک من محسن Mohsen و اسم خانوادگى من بنان
> Banan است.
> 
> Now look at the display of citation line above
> starting on the left with ' >>'  after the 
>   >> For example when I write:
> 
> You see how the Farsi piece (word sequence -- not
> character sequence) is flipped around "Mohsen".

Which one of the two is correct, in the original text or in the
citations?  (I don't read Farsi.)

I think the issue here is paragraph direction, which in Emacs is
dynamically determined by default.  See below.

> In the browser my original text appears as it does
> in the citation.

Does the browser display it flushed to the left or to the right?

Does it help to say "M-x set-variable RET bidi-paragraph-direction RET
right-to-left RET" in Emacs?  Do you see both versions of the text
correctly then?

> And it displays just fine under emacs24 but word
> sequences are flipped when the browser renders
> that same byte sequence.

So the Emacs display is correct, is that what you are saying?  I don't
care about Firefox, but I do care about the bidirectional display in
Emacs.


_______________________________________________
emacs-bidi mailing list
emacs-bidi@gnu.org
https://lists.gnu.org/mailman/listinfo/emacs-bidi

Now: Paragraph Direction Detection and Harmonization -- Was: Re: Bidirectional editing in Emacs -- main design decisions

[Mohsen BANAN]

>>>>> On Mon, 25 Apr 2011 21:59:43 +0300, Eli Zaretskii <eliz@gnu.org> said:
  >> You see how the Farsi piece (word sequence -- not
  >> character sequence) is flipped around "Mohsen".

  Eli> Which one of the two is correct, in the original text or in the
  Eli> citations?  (I don't read Farsi.)

Of course, the original is correct.
Because I wrote it and I wrote it with emacs.

  Eli> I think the issue here is paragraph direction, which in Emacs is
  Eli> dynamically determined by default.  See below.

I agree. That is the problem.  

I have therefore changed the subject line to:
 Paragraph Direction Detection and Harmonization

Please see below as I expand on what parts I see
as non-problem and what parts I think can be
improved.

  >> In the browser my original text appears as it does
  >> in the citation.

  Eli> Does the browser display it flushed to the left or to the right?

Everything is flushed to the left. Which is
consistent your diagnosis of paragraph direction problem.

  Eli> Does it help to say "M-x set-variable RET bidi-paragraph-direction RET
  Eli> right-to-left RET" in Emacs?  Do you see both versions of the text
  Eli> correctly then?

Yes.

I have added key bindings for myself for
 (set-variable  'bidi-paragraph-direction 'right-to-left)
and 
 (set-variable  'bidi-paragraph-direction 'left-to-right)
for debugging. 

  >> And it displays just fine under emacs24 but word
  >> sequences are flipped when the browser renders
  >> that same byte sequence.

  Eli> So the Emacs display is correct, is that
  Eli> what you are saying?  

I am saying that emacs display is correct but that
there are interoperability problems.

  Eli> I don't care about Firefox, but I do care
  Eli> about the bidirectional display in Emacs.

The nature of bidi text is such that it demands
harmony.

For example, I think that it is worthwhile for
emacs24 to have a good Conformance Statement for 
 http://unicode.org/reports/tr9/

The existence of -- Unicode Standard Annex #9 --
Unicode Bidirectional Algorithm -- speaks to that
requirement for harmonization.

I looked in there for information about 
dynamic paragraph direction detection and did not
find much.

I think that the dynamic paragraph direction
detection in emacs needs improvement.

In the email citation the direction was guessed
wrong although that paragraph was dominantly RTL.

Can you please expand on what algorithm you use to 
determine dynamic paragraph direction?

With respect to:

  Eli> I don't care about Firefox

I have a different view. I see 
Emacs and Firefox as joint sisters.

See

   Blee: A Deeply Integrated Emacs User Environment For the Software-Service Continuum
   http://linuxfestnorthwest.org/sessions/blee-deeply-integrated-emacs-user-environment-software-service-continum

for that perspective. 

A lot of benefits come from integrating the
browser with emacs.

With respect to Mozilla-Emacs display
inconsistency for bidi email generated with emacs,
I think the solution is to generate html and
specify paragraph direction explicitly in html.

Beyond the basic bidi capability in emacs, there
are several layers above it that we now need to
cultivate.

Thanks for your help.

...Mohsen
...محسن

Re: Now: Paragraph Direction Detection and Harmonization -- Was: Re: Bidirectional editing in Emacs -- main design decisions

[Eli Zaretskii]

> From: Mohsen BANAN <list-general@mohsen.1.banan.byname.net>
> Cc: Mohsen BANAN <list-general@mohsen.1.banan.byname.net>,  emacs-bidi@gnu.org,  emacs-devel@gnu.org
> Date: Mon, 25 Apr 2011 14:31:22 -0700
> 
> I am saying that emacs display is correct but that
> there are interoperability problems.

That could be, but it sounds like the solution to those problems
should be on the Firefox side.  Or maybe Firefox also has some
customization feature, like bidi-paragraph-direction in Emacs.

> For example, I think that it is worthwhile for
> emacs24 to have a good Conformance Statement for 
>  http://unicode.org/reports/tr9/

We already do, see etc/NEWS:

  Reordering of bidirectional text for display in Emacs is a "Full
  bidirectionality" class implementation of the Unicode Bidirectional
  Algorithm.

Is that what you meant by "conformance statement"?  If not, what is
it?

> The existence of -- Unicode Standard Annex #9 --
> Unicode Bidirectional Algorithm -- speaks to that
> requirement for harmonization.

If every application out there implements UAX#9 to the letter, there
shouldn't be interoperability problems.

> I looked in there for information about 
> dynamic paragraph direction detection and did not
> find much.

It is in section 3.3.1 there:

  P2. In each paragraph, find the first character of type L, AL, or R.
  [...]
  P3. If a character is found in P2 and it is of type AL or R, then set
  the paragraph embedding level to one; otherwise, set it to zero.

  Whenever a higher-level protocol specifies the paragraph level, rules
  P2 and P3 do not apply.

Setting the paragraph embedding level to 1 means the paragraph
direction is right-to-left; zero means it is left-to-right.

The last sentence, about "higher-level protocols", is important,
because Emacs does employ such protocols.

> I think that the dynamic paragraph direction
> detection in emacs needs improvement.

If you have specific practical suggestions (after reading the
description I point to below), please spell them out.

> In the email citation the direction was guessed
> wrong although that paragraph was dominantly RTL.

That is because there was no paragraph separator between the previous
parts of text, which was left-to-right, and the Farsi part.  Emacs
generally keeps the direction of the previous paragraph as long as it
doesn't see a new one.

> Can you please expand on what algorithm you use to 
> determine dynamic paragraph direction?

I explained that some time ago, see this message:

  http://lists.gnu.org/archive/html/emacs-devel/2009-10/msg00234.html

Look for "Paragraph base direction" there.  In addition, the Emacs
manual tells something about this in the node "Bidirectional Editing".

> With respect to:
> 
>   Eli> I don't care about Firefox
> 
> I have a different view. I see 
> Emacs and Firefox as joint sisters.

Well, "don't care" means it's not my job to make Firefox better.  A
single individual can only do this much.  I will be happy enough if
bidirectional editing support in Emacs is good enough, even though
other applications will need to catch up.

> With respect to Mozilla-Emacs display
> inconsistency for bidi email generated with emacs,
> I think the solution is to generate html and
> specify paragraph direction explicitly in html.

There's no need for that in Emacs, see the message in the URL above.
I explained there how you can control the base paragraph direction in
plain text, on the user level, by inserting special control
characters.

> Beyond the basic bidi capability in emacs, there
> are several layers above it that we now need to
> cultivate.

Yes.  Volunteers are welcome to contribute code to that effect.

Now: Paragraph Direction Detection and Harmonization -- Was: Re: Bidirectional editing in Emacs -- main design decisions

[Stephen J. Turnbull]

Mohsen BANAN writes:

 >   Eli> So the Emacs display is correct, is that
 >   Eli> what you are saying?  
 > 
 > I am saying that emacs display is correct

Oh, why is there no period here?

 > but that there are interoperability problems.

Please, let's not go there.  We know one good path (not perfect, but
good) to interoperability: Implement UTR#9.  The other paths are the
road to interoperability hell, most likely.

So it's too bad for the users of the other application, then, unless
they're programmers capable of hacking the code themselves.  As a
20-year veteran of the Japanese interoperability disaster, which was
two decades in progress when I joined it and which continues to this
very day, please, have patience, for five years if it takes that long.
(< 5 40) => t!

The thing is, every concession to broken implementations delays true
harmony (by which I mean "reliable interoperability", not "pedantic
standards conformance").  Not to mention strengthening proprietary
software, which often seeks to exploit the "tippiness" induced by
interoperability problems.

 > The nature of bidi text is such that it demands
 > harmony.

True, but overly specific.  s/bidi//  ;-)  Everybody thinks their
problem is special.  Of course it is in one way, but in another, it's
not.

 >   Eli> I don't care about Firefox
 > 
 > I have a different view. I see Emacs and Firefox as joint sisters.

Of course users care.  But user wishes don't generate code.  Eli cares
about users; he's proved that amply over the years.  So if he says he
"doesn't care", that's not just making obscene gestures at the users,
that's his expert judgment that it's not worth doing.  He may not be
able to "prove" it, but his expertise is proven to me by 2 decades of
acquaintance.

If you have the ability to code, Eli will help.  I can testify to
that, too.  But "don't care" means Eli isn't going to do it himself,
it's too much work compared to the gains possible by working on other
areas.

And it's not like the Firefox developers don't care, either.  It would
be much better to concentrate on lobbying them.  They *want* to be
correct, and will be made happier once Firefox conforms to the
standard.

 > I think the solution is to generate html and
 > specify paragraph direction explicitly in html.

That's OK IMO, although somewhat undesirable.  (It will slow the
adoption of correct and full implementations of the bidi algorithm.)

That's not an Emacs problem, though.  That's an MUA problem.  I guess
you're talking to the right person on that, since RMail is still the
official Emacs MUA, and Eli uses and (sometimes, at least) contributes
to RMail.  But if you want other people, who use a wide variety of
MUAs, to do it that way, you'll need to talk to larsi and/or Ted
Zlatanov (Gnus/message-mode) and Uday Reddy (VM), at least.

 > Beyond the basic bidi capability in emacs, there
 > are several layers above it that we now need to
 > cultivate.

+1 on that, though.  I hope Eli will focus on that, personally.  (But
that's 'cause I'm a standards geek, so you can multiply that by about
0.1 importance factor -- I have no personal need for bidi, or
experience with it. Someday ... :-) 

Re: Now: Paragraph Direction Detection and Harmonization -- Was: Re: Bidirectional editing in Emacs -- main design decisions

[Mohsen BANAN]

>>>>> On Tue, 26 Apr 2011 01:00:52 +0300, Eli Zaretskii <eliz@gnu.org> said:

  >> Beyond the basic bidi capability in emacs, there
  >> are several layers above it that we now need to
  >> cultivate.

  Eli> Yes.  Volunteers are welcome to contribute code to that effect.

The currency for respect around here appears to be
code. Let me attempt to address that first.

Please permit me to volunteer for a few things.

The common thread in all of these will be to make
use of emacs for perso-arabic script easier and
more common.

1) A transliteration emacs input method for
   Persian/Farsi.

   I'll be sending out the quail-persian-translit.el
   file right after this. AFAIK there is no input 
   method for Perisan/Farsi in emacs24. With bidi
   in place, we now need that.

2) A Perisan/Farsi keyboard input method for emacs.

   I'll be sending out the quail-persian-isiri9147.el
   file right after this. This is based on the 
   latest Iranian standard for keyboard layouts.

3) A document titled:

     Perso-Arabic Script In the Halaal and Convivial Quadrant.

     نگارِش به فارسي و عربى در چهارچوب حلال و با
     هم زندِگاني.
  
   Halaal Software is consistent with Libre/Free
   Software but goes beyond that.

   Convivial is not used in its English form.  It
   is used in Globish as introduced by Ivan Illich 
   in "Tools for Conviviality".

   The purpose of this document is to bring all
   necessary pieces (software, fonts, dictionaries,
   knowledge) together for perso-arabic script in
   the context of debian+emacs+xetex.

   A very preliminary draft is available at:
        http://mohsen.1.banan.byname.net/PLPC/180003
        http://mohsen.1.banan.byname.net/content/generated/doc.free/bystar/PLPC/180003/current/presentationEn.pdf

There is more emacs related stuff in the works
that is not quite ready for exposure yet.

Regards,

...Mohsen

Re: Now: Paragraph Direction Detection and Harmonization -- Was: Re: Bidirectional editing in Emacs -- main design decisions

[Eli Zaretskii]

> From: Mohsen BANAN <list-general@mohsen.1.banan.byname.net>
> Cc: Mohsen BANAN <list-general@mohsen.1.banan.byname.net>,  emacs-bidi@gnu.org,  emacs-devel@gnu.org
> Date: Tue, 26 Apr 2011 00:56:12 -0700
> 
> 
> 3) A document titled:
> 
>      Perso-Arabic Script In the Halaal and Convivial Quadrant.

I'm not sure this document should be included in Emacs, but I'll leave
it to Chong and Stefan to decide on that.

Re: Now: Paragraph Direction Detection and Harmonization

[Mohsen BANAN]

>>>>> On Tue, 26 Apr 2011 01:00:52 +0300, Eli Zaretskii <eliz@gnu.org> said:

  >> For example, I think that it is worthwhile for
  >> emacs24 to have a good Conformance Statement for 
  >> http://unicode.org/reports/tr9/

  Eli> We already do, see etc/NEWS:

  Eli>   Reordering of bidirectional text for display in Emacs is a "Full
  Eli>   bidirectionality" class implementation of the Unicode Bidirectional
  Eli>   Algorithm.

  Eli> Is that what you meant by "conformance statement"?  If not, what is
  Eli> it?

By a good Conformance Statement, I meant more than
that.

I meant it in the context of ISO 9646 and concepts
of Protocol Implementation Conformance Statement
(PICS) Proforma and PICS.

To evaluate conformance of a particular
implementation, it is necessary to have a
statement of the capabilities and options that
have been implemented for a given protocol (and
bidi can reasonably be regarded a protocol, as it
is based on expected behavior).  Such
a statement is called a Protocol Implementation
Conformance Statement (PICS).

I don't know if there is a PICS-Proforma for 
http://unicode.org/reports/tr9/.

If there was, it would take the form of tables
refering to the base spec asking for details of
implementation with regard to each
section/option/feature/...  of the specification.

An example of PICS-Proforma in the RFC Series is:
http://tools.ietf.org/html/rfc1708

Pics is taken more seriously in the ISO/IEC
ITU/CCITT cultures. Various people have various
opinions on merits of the PICS model. I like the
PICS model and have managed to put it to good use
in realization of large networks.

In the absence of a PICS-Proforma, based on your
implementation details, you (or somebody) can go
through the specification and include emacs-bidi
implementation details and cross-referencing UAX#9
sections.

Given the lack of precision for uniformity in
determination of say paragraph direction, based on
the emacs PICS, we can then visit our sisters at
Mozilla and have a conversation towards more
harmony.

...Mohsen

Re: Now: Paragraph Direction Detection and Harmonization

[Eli Zaretskii]

> From: Mohsen BANAN <list-general@mohsen.1.banan.byname.net>
> Cc: Mohsen BANAN <list-general@mohsen.1.banan.byname.net>,  emacs-bidi@gnu.org,  emacs-devel@gnu.org
> Date: Tue, 26 Apr 2011 11:24:04 -0700
> 
> I don't know if there is a PICS-Proforma for 
> http://unicode.org/reports/tr9/.
> 
> If there was, it would take the form of tables
> refering to the base spec asking for details of
> implementation with regard to each
> section/option/feature/...  of the specification.
> 
> An example of PICS-Proforma in the RFC Series is:
> http://tools.ietf.org/html/rfc1708

I don't think such a thing exists for UAX#9.  There's Section 4
"Bidirectional Conformance"; Emacs implements everything it requires
under "Full bidirectionality" class, as stated in the fragment of NEWS
I cited.  That's all there is to it.

> In the absence of a PICS-Proforma, based on your
> implementation details, you (or somebody) can go
> through the specification and include emacs-bidi
> implementation details and cross-referencing UAX#9
> sections.

The source code in bidi.c already includes such cross-references.

> Given the lack of precision for uniformity in
> determination of say paragraph direction, based on
> the emacs PICS, we can then visit our sisters at
> Mozilla and have a conversation towards more
> harmony.

The algorithm used by Emacs to determine the paragraph base direction
is exactly as prescribed by UAX#9.  What differs, under the
"high-level protocol" fire escape, is the definition of the
paragraph.  This definition is explained in the message I quoted
earlier in this thread and in the Emacs User Manual.  It is quite
simple and easy to understand, so I really see no place for any
misunderstanding.

Re: Now: Paragraph Direction Detection and Harmonization

[Mohsen BANAN]

>>>>> On Tue, 26 Apr 2011 21:05:40 +0300, Eli Zaretskii <eliz@gnu.org> said:

  >> From: Mohsen BANAN <list-general@mohsen.1.banan.byname.net>
  >> Cc: Mohsen BANAN <list-general@mohsen.1.banan.byname.net>,  emacs-bidi@gnu.org,  emacs-devel@gnu.org
  >> Date: Tue, 26 Apr 2011 00:56:12 -0700
  >> 
  >> 
  >> 3) A document titled:
  >> 
  >> Perso-Arabic Script In the Halaal and Convivial Quadrant.

  Eli> I'm not sure this document should be included in Emacs, but I'll leave
  Eli> it to Chong and Stefan to decide on that.

It does not make good sense to include that
document in Emacs.

Its scope is broader than Emacs.

It is an Emacs related resource that I intend to
maintain. I mentioed it as it related to the
subject matter at hand.

...Mohsen

Requesting instructions for enabling bidi by default

[Mohsen BANAN]

>>>>> On Mon, 25 Apr 2011 20:58:26 +0300, Eli Zaretskii <eliz@gnu.org> said:

  Eli> Did you (setq bidi-display-reordering t) in the buffer where this is
  Eli> displayed?  Without that, the bidirectional reordering does not
  Eli> happen.  (Some day, this will be on by default, but not yet.)

Would you please teach me on how to make 
 (setq bidi-display-reordering t) 
default for all buffers always.

I can build from sources and if you were to point
me to the relevant files and relevant changes, I
can take it from there.

I am in the process of including farsi as an
integral part of everything I do (gnus, bbdb,
calendar, emms, auctex, dict, ...) but even with
good keybindings (setq bidi-display-reordering t)
-- including in the mini-buffer,
e.g. (ispell-word) -- is a hassle ...

I do understand your reasoning for deferring that
for later and considerations for mainstream
stability.

The likes of me can profit from instructions for
having bidi enabled everywhere and always in the
interim.

Thanks in advance.

...Mohsen

Re: Requesting instructions for enabling bidi by default

[Juanma Barranquero]

On Thu, Apr 28, 2011 at 02:52, Mohsen BANAN
<list-general@mohsen.1.banan.byname.net> wrote:

> Would you please teach me on how to make
>  (setq bidi-display-reordering t)
> default for all buffers always.

Adding

(setq-default bidi-display-reordering t)

to your .emacs should be enough.

    Juanma

_______________________________________________
emacs-bidi mailing list
emacs-bidi@gnu.org
https://lists.gnu.org/mailman/listinfo/emacs-bidi