Re: Now: Paragraph Direction Detection and Harmonization -- Was: Re: Bidirectional editing in Emacs -- main design decisions

[Eli Zaretskii <eliz@gnu.org>]

> From: Mohsen BANAN <list-general@mohsen.1.banan.byname.net>
> Cc: Mohsen BANAN <list-general@mohsen.1.banan.byname.net>,  emacs-bidi@gnu.org,  emacs-devel@gnu.org
> Date: Mon, 25 Apr 2011 14:31:22 -0700
> 
> I am saying that emacs display is correct but that
> there are interoperability problems.

That could be, but it sounds like the solution to those problems
should be on the Firefox side.  Or maybe Firefox also has some
customization feature, like bidi-paragraph-direction in Emacs.

> For example, I think that it is worthwhile for
> emacs24 to have a good Conformance Statement for 
>  http://unicode.org/reports/tr9/

We already do, see etc/NEWS:

  Reordering of bidirectional text for display in Emacs is a "Full
  bidirectionality" class implementation of the Unicode Bidirectional
  Algorithm.

Is that what you meant by "conformance statement"?  If not, what is
it?

> The existence of -- Unicode Standard Annex #9 --
> Unicode Bidirectional Algorithm -- speaks to that
> requirement for harmonization.

If every application out there implements UAX#9 to the letter, there
shouldn't be interoperability problems.

> I looked in there for information about 
> dynamic paragraph direction detection and did not
> find much.

It is in section 3.3.1 there:

  P2. In each paragraph, find the first character of type L, AL, or R.
  [...]
  P3. If a character is found in P2 and it is of type AL or R, then set
  the paragraph embedding level to one; otherwise, set it to zero.

  Whenever a higher-level protocol specifies the paragraph level, rules
  P2 and P3 do not apply.

Setting the paragraph embedding level to 1 means the paragraph
direction is right-to-left; zero means it is left-to-right.

The last sentence, about "higher-level protocols", is important,
because Emacs does employ such protocols.

> I think that the dynamic paragraph direction
> detection in emacs needs improvement.

If you have specific practical suggestions (after reading the
description I point to below), please spell them out.

> In the email citation the direction was guessed
> wrong although that paragraph was dominantly RTL.

That is because there was no paragraph separator between the previous
parts of text, which was left-to-right, and the Farsi part.  Emacs
generally keeps the direction of the previous paragraph as long as it
doesn't see a new one.

> Can you please expand on what algorithm you use to 
> determine dynamic paragraph direction?

I explained that some time ago, see this message:

  http://lists.gnu.org/archive/html/emacs-devel/2009-10/msg00234.html

Look for "Paragraph base direction" there.  In addition, the Emacs
manual tells something about this in the node "Bidirectional Editing".

> With respect to:
> 
>   Eli> I don't care about Firefox
> 
> I have a different view. I see 
> Emacs and Firefox as joint sisters.

Well, "don't care" means it's not my job to make Firefox better.  A
single individual can only do this much.  I will be happy enough if
bidirectional editing support in Emacs is good enough, even though
other applications will need to catch up.

> With respect to Mozilla-Emacs display
> inconsistency for bidi email generated with emacs,
> I think the solution is to generate html and
> specify paragraph direction explicitly in html.

There's no need for that in Emacs, see the message in the URL above.
I explained there how you can control the base paragraph direction in
plain text, on the user level, by inserting special control
characters.

> Beyond the basic bidi capability in emacs, there
> are several layers above it that we now need to
> cultivate.

Yes.  Volunteers are welcome to contribute code to that effect.