Re: Message Mode and bidi

[Eli Zaretskii <eliz@gnu.org>]

> From: Eric Abrahamsen <eric@ericabrahamsen.net>
> Date: Mon, 19 Feb 2024 21:16:51 -0800
> 
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Eric Abrahamsen <eric@ericabrahamsen.net>
> >> Date: Mon, 19 Feb 2024 16:46:22 -0800
> >> 
> >> Christopher Culver via "Emacs development discussions."
> >> <emacs-devel@gnu.org> writes:
> >> 
> >> > Joost Kremers <joostkremers@fastmail.fm> writes:
> >> >> When you compose a new message, is there a line "--text follows this line--"
> >> >> separating the headers and the message text? In my case, there is (I use mu4e)
> >> >> and when I type Arabic text on the line below this text, I get the effect you
> >> >> mention. If I leave an empty line after "--text follows this line--", bidi works
> >> >> as expected.
> >> >
> >> > Indeed, if I just go down one line and then begin typing, bidi works as
> >> > expected. I am feeling very foolish that I did not even try this. Thank
> >> > you for clearing up this problem, and for shedding light on how Emacs
> >> > considers paragraphs.
> >> 
> >> This is a bit weird, because the value of `mail-header-separator'
> >> ("--text follows this line--") is added to both `paragraph-start' and
> >> `paragraph-separate') in `message-mode'. You'd think one of those would
> >> do it.
> >
> > Emacs doesn't use paragraph-separate and paragraph-start to define
> > where a paragraph starts and ends, for the purposes of determining the
> > base directionality of a paragraph.  It uses separate variables for
> > that, see the node "Bidirectional Editing" in the Emacs user manual.
> > The reason for using separate variables is because several modes,
> > including (but not limited to) message-mode, set the former variables
> > to regexps that get in the way of bidi reordering, and could easily
> > produce wrong results on display.
> 
> Do you think we'd stand a chance of finding values for
> bidi-paragraph-start|separate-re that would resolve this particular
> issue?

We already have those values described in the Emacs manual:

     Each paragraph of bidirectional text can have its own “base
  direction”, either right-to-left or left-to-right.  Text in
  left-to-right paragraphs begins on the screen at the left margin of the
  window and is truncated or continued when it reaches the right margin.
  By contrast, text in right-to-left paragraphs is displayed starting at
  the right margin and is continued or truncated at the left margin.  By
  default, paragraph boundaries are empty lines, i.e., lines consisting
  entirely of whitespace characters.  To change that, you can customize
  the two variables ‘bidi-paragraph-start-re’ and
  ‘bidi-paragraph-separate-re’, whose values should be regular expressions
  (strings); e.g., to have a single newline start a new paragraph, set
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  both of these variables to ‘"^"’.  These two variables are buffer-local
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  (*note Locals::).

But beware: doing this in Emacs will cause effects that are unpleasant
to readers of bidirectional text, because you could have "chess-like"
text display, like this:

asasasasasasasasasasasasassa
                                                ASASASASASASASASASASASASA
xcxcxcxcxcxcxcxcxcxcxcxcxc
                                                       JKJNJKKKNKNKNKNKNK

etc.  Here upper-case letters stand for RTL (like Arabic or Farsi)
text and lower-case letters stand for LTR (like Latin or Cyrillic)
text.

Let me explain.  UBA, the Unicode Bidirectional Algorithm which Emacs
implements, was designed for text-processing programs that wrap long
lines without inserting hard newlines.  For those applications, a
single hard newline indicates a new paragraph, and thus recalculating
the base direction of a paragraph after a newline is justified.

But in Emacs, we have a lot of text filled and wrapped using hard
newlines, and filling can insert a newline in an arbitrary place
inside the paragraph.  If it happens that a newline was inserted
before a strong right-to-left character, under the strict UBA rules
the next line will be considered as a paragraph with right-to-left
base direction, and rendered starting at the right window edge.  Which
leads to the above "chess-like" display.  Since that is basically
unacceptable for Emacs users, we require at least one empty line to
separate paragraphs.  The price of a single empty line before the body
of an email message that needs to be rendered right to left is a small
price to pay for solving the horrible display effect of changing the
paragraph's base direction after each newline.  An alternative would
be to use visual-line-mode instead of wrapping using hard newlines,
but that is still relatively rare in Emacs, especially in email
messages.