Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii <eliz@gnu.org>]

> From: Mohsen BANAN <list-general@mohsen.1.banan.byname.net>
> Cc: Mohsen BANAN <list-general@mohsen.1.banan.byname.net>,  emacs-devel@gnu.org
> Date: Tue, 02 Aug 2011 19:39:08 -0700
> 
>   >>>
>   >>> This is paragraph 1 all in english.
>   >>>
>   >>> اين پاراگراف ۲ تماما به فارسى است.
>   >>>
>   >>> This is paragraph 3 which is mixed with فارسى and english.
>   >>>
>   >>> اين پاراگراف ۴ مخلوت English و فارسى است.
>   >>>
> 
> ---
> 
> Note all 4 paragraphs' directionality are detected
> fine based on UAX#9 prior to citation.
> 
> Once cited/prefixed they all become 1 paragraph.
> 
> The communication ramifications on paragraph 2 are
> minor. The reader would get it. 
> 
> But communication ramifications on paragraph 4 are
> very problematic. The sentence becomes
> incomprehensible/mangled.

Yes, I agree.

> So, here is what I am suggesting:
> 
>  - We add an additional value of 'fancy to
>    bidi-paragraph-direction making it:
>     nil, left-to-right, right-to-left, fancy
> 
>  - If bidi-paragraph-direction is 'fancy,
>     
>      - do prefix guessing
>      - ignore prefix
>      - do UAX#9
>      - re-insert prefix
> 
> This will likely help with simple tabularization
> in addition to mail citations.
> 
> What do you think?

This is too complicated.  Paragraph detection is at the lowest level
of the reordering engine, so it must be simple and fast.  We saw just
yesterday a case of severe slowdown of redisplay because paragraph
detection was working too hard.

I have an idea that would be much easier to implement: ignore neutral
and weak characters for the purposes of the decision where a paragraph
ends.  This can be done by modifying the regexp used to detect the
paragraph separator.  This would make all the citations preserve their
original paragraphs, notwithstanding the citation.  Then it will look
like this (but without the added empty lines, of course):

  >>>
  >>> This is paragraph 1 all in english.
  >>>

  >>> اين پاراگراف ۲ تماما به فارسى است.
  >>>

  >>> This is paragraph 3 which is mixed with فارسى and english.
  >>>

  >>> اين پاراگراف ۴ مخلوت English و فارسى است.
  >>>

(I assume you are looking at this in Emacs 24, which will reorder
correctly.)  Is this okay?

Anyway, this feature will probably have to wait until Emacs 24.2,
because there are more serious issues on the table, and we want to
start the pretest for Emacs 24.1 ASAP.  Modifications of paragraph
separator is something that needs to be thoroughly tested before it
hits the end users' machines, because it can potentially destabilize
the display engine to a great extent.

For now, users will have to learn to insert empty lines ;-)