unofficial mirror of emacs-devel@gnu.org 
 help / color / mirror / code / Atom feed
* Message Mode and bidi
@ 2024-02-19  1:12 Christopher Culver via Emacs development discussions.
  2024-02-19  3:36 ` Eli Zaretskii
  0 siblings, 1 reply; 14+ messages in thread
From: Christopher Culver via Emacs development discussions. @ 2024-02-19  1:12 UTC (permalink / raw)
  To: emacs-devel

Modes like text-mode and Org mode now have good support for
bidirectional text. When one switches to an input method for a
right-to-left language, like farsi-translit-banan, and types a letter,
the text is immediately aligned to the right side of the window and the
cursor appears correctly to the left of the typed text.

This functionality is missing from Message Mode, as I find when writing
a new e-mail in Gnus. Right-to-left text can be inserted there, but the
text is not aligned to the right side of the window, and the cursor
incorrectly appears to the right of the typed text.

Thus my question is: is this a bug in Message Mode that could be
reported, i.e. the required functionality has not been implemented there
as in other popular modes?  Or does this lack of full bidi support in
this mode represent a deliberate choice on the part of Emacs devs?



^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Message Mode and bidi
  2024-02-19  1:12 Message Mode and bidi Christopher Culver via Emacs development discussions.
@ 2024-02-19  3:36 ` Eli Zaretskii
  2024-02-19 21:29   ` Christopher Culver via Emacs development discussions.
  0 siblings, 1 reply; 14+ messages in thread
From: Eli Zaretskii @ 2024-02-19  3:36 UTC (permalink / raw)
  To: Christopher Culver; +Cc: emacs-devel

> Date: Mon, 19 Feb 2024 02:12:11 +0100
> From:  Christopher Culver via "Emacs development discussions." <emacs-devel@gnu.org>
> 
> Modes like text-mode and Org mode now have good support for
> bidirectional text. When one switches to an input method for a
> right-to-left language, like farsi-translit-banan, and types a letter,
> the text is immediately aligned to the right side of the window and the
> cursor appears correctly to the left of the typed text.
> 
> This functionality is missing from Message Mode, as I find when writing
> a new e-mail in Gnus. Right-to-left text can be inserted there, but the
> text is not aligned to the right side of the window, and the cursor
> incorrectly appears to the right of the typed text.

I cannot reproduce this.  What I see is that Message mode supports
bidirectional text as expected, including what you describe in the
previous paragraph regarding right-aligning the RTL text.  So please
tell more what you did by describing in detail the commands you typed,
so we could understand why you think bidirectional text is not
sufficiently well supported by Message mode.  I expect i t is due to
some misunderstanding of how Emacs implements and supports bidi,
perhaps related to the Emacs interpretation of "paragraph" and
"paragraph direction".

> Thus my question is: is this a bug in Message Mode that could be
> reported, i.e. the required functionality has not been implemented there
> as in other popular modes?  Or does this lack of full bidi support in
> this mode represent a deliberate choice on the part of Emacs devs?

There's no bug that I could see.  I think it is some mistake on your
part, which will be revealed if you describe the steps which led you
to the above conclusion.



^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Message Mode and bidi
  2024-02-19  3:36 ` Eli Zaretskii
@ 2024-02-19 21:29   ` Christopher Culver via Emacs development discussions.
  2024-02-19 21:50     ` Joost Kremers
  0 siblings, 1 reply; 14+ messages in thread
From: Christopher Culver via Emacs development discussions. @ 2024-02-19 21:29 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: emacs-devel

Eli Zaretskii <eliz@gnu.org> writes:
> I cannot reproduce this.  What I see is that Message mode supports
> bidirectional text as expected, including what you describe in the
> previous paragraph regarding right-aligning the RTL text.  So please
> tell more what you did by describing in detail the commands you typed.

I have tried this with a completely virgin Emacs installation, but
Message Mode continues to show less than full support for bidi text. The
steps were as follows:

* Move .emacs.d directory and .gnus.el[c] files to some other location.

* Launch Emacs.

* M-x gnus

* Gnus will pop up an error about unconfigured nntp support, but ignore.

* Press a in order to begin composing a new email in Message Mode.

* Move the cursor down to the message-body section, switch input method
  to farsi-translit-banan, and begin typing. The text does not
  automatically align to the right side of the window, nor does the
  cursor appear correctly to the left of the text being
  entered. (However, I do get full bidi support in a new .txt file
  created in text-mode.)

I am using Emacs 29.1 as packaged by Debian Stable Backports
(stable-bpo) on Wayland, so presumably the pure-gtk UI.

Because email is a special case, with decades-old standards, I thought I
should ask the devs if this phenomenon represented a deliberate choice
on the part of the Emacs devs. If this is indeed a bug, then please let
me know whether it should be reported to Debian as the packager, or if
it is something to be resolved by the Emacs devs themselves.



^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Message Mode and bidi
  2024-02-19 21:29   ` Christopher Culver via Emacs development discussions.
@ 2024-02-19 21:50     ` Joost Kremers
  2024-02-19 22:06       ` Christopher Culver via Emacs development discussions.
  2024-02-20  3:29       ` Eli Zaretskii
  0 siblings, 2 replies; 14+ messages in thread
From: Joost Kremers @ 2024-02-19 21:50 UTC (permalink / raw)
  To: Christopher Culver; +Cc: Eli Zaretskii, emacs-devel


On Mon, Feb 19 2024, Christopher Culver via "Emacs development discussions." wrote:
> Eli Zaretskii <eliz@gnu.org> writes:
>> I cannot reproduce this.  What I see is that Message mode supports
>> bidirectional text as expected, including what you describe in the
>> previous paragraph regarding right-aligning the RTL text.  So please
>> tell more what you did by describing in detail the commands you typed.
>
> I have tried this with a completely virgin Emacs installation, but
> Message Mode continues to show less than full support for bidi text.

When you compose a new message, is there a line "--text follows this line--"
separating the headers and the message text? In my case, there is (I use mu4e)
and when I type Arabic text on the line below this text, I get the effect you
mention. If I leave an empty line after "--text follows this line--", bidi works
as expected.

I believe this is due to the interaction of two things: 1. Emacs uses the first
word of a paragraph to determine whether the paragraph is ltr or rtl; and 2.
Emacs assumes that a paragraph is preceded by an empty line. So if you type rtl
text on the line below the "--text follows this line--" separator, Emacs
assumes the separator and the text you're typing belong to the same paragraph,
and since the first word of this paragraph is from an ltr writing system, the
paragraph is displayed ltr.



-- 
Joost Kremers
Life has its moments



^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Message Mode and bidi
  2024-02-19 21:50     ` Joost Kremers
@ 2024-02-19 22:06       ` Christopher Culver via Emacs development discussions.
  2024-02-20  0:46         ` Eric Abrahamsen
  2024-02-20  3:31         ` Eli Zaretskii
  2024-02-20  3:29       ` Eli Zaretskii
  1 sibling, 2 replies; 14+ messages in thread
From: Christopher Culver via Emacs development discussions. @ 2024-02-19 22:06 UTC (permalink / raw)
  To: Joost Kremers; +Cc: emacs-devel

Joost Kremers <joostkremers@fastmail.fm> writes:
> When you compose a new message, is there a line "--text follows this line--"
> separating the headers and the message text? In my case, there is (I use mu4e)
> and when I type Arabic text on the line below this text, I get the effect you
> mention. If I leave an empty line after "--text follows this line--", bidi works
> as expected.

Indeed, if I just go down one line and then begin typing, bidi works as
expected. I am feeling very foolish that I did not even try this. Thank
you for clearing up this problem, and for shedding light on how Emacs
considers paragraphs.



^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Message Mode and bidi
  2024-02-19 22:06       ` Christopher Culver via Emacs development discussions.
@ 2024-02-20  0:46         ` Eric Abrahamsen
  2024-02-20  3:37           ` Eli Zaretskii
  2024-02-20  3:31         ` Eli Zaretskii
  1 sibling, 1 reply; 14+ messages in thread
From: Eric Abrahamsen @ 2024-02-20  0:46 UTC (permalink / raw)
  To: emacs-devel

Christopher Culver via "Emacs development discussions."
<emacs-devel@gnu.org> writes:

> Joost Kremers <joostkremers@fastmail.fm> writes:
>> When you compose a new message, is there a line "--text follows this line--"
>> separating the headers and the message text? In my case, there is (I use mu4e)
>> and when I type Arabic text on the line below this text, I get the effect you
>> mention. If I leave an empty line after "--text follows this line--", bidi works
>> as expected.
>
> Indeed, if I just go down one line and then begin typing, bidi works as
> expected. I am feeling very foolish that I did not even try this. Thank
> you for clearing up this problem, and for shedding light on how Emacs
> considers paragraphs.

This is a bit weird, because the value of `mail-header-separator'
("--text follows this line--") is added to both `paragraph-start' and
`paragraph-separate') in `message-mode'. You'd think one of those would
do it.




^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Message Mode and bidi
  2024-02-19 21:50     ` Joost Kremers
  2024-02-19 22:06       ` Christopher Culver via Emacs development discussions.
@ 2024-02-20  3:29       ` Eli Zaretskii
  1 sibling, 0 replies; 14+ messages in thread
From: Eli Zaretskii @ 2024-02-20  3:29 UTC (permalink / raw)
  To: Joost Kremers; +Cc: crculver, emacs-devel

> From: Joost Kremers <joostkremers@fastmail.fm>
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> Date: Mon, 19 Feb 2024 22:50:50 +0100
> 
> On Mon, Feb 19 2024, Christopher Culver via "Emacs development discussions." wrote:
> > Eli Zaretskii <eliz@gnu.org> writes:
> >> I cannot reproduce this.  What I see is that Message mode supports
> >> bidirectional text as expected, including what you describe in the
> >> previous paragraph regarding right-aligning the RTL text.  So please
> >> tell more what you did by describing in detail the commands you typed.
> >
> > I have tried this with a completely virgin Emacs installation, but
> > Message Mode continues to show less than full support for bidi text.
> 
> When you compose a new message, is there a line "--text follows this line--"
> separating the headers and the message text? In my case, there is (I use mu4e)
> and when I type Arabic text on the line below this text, I get the effect you
> mention. If I leave an empty line after "--text follows this line--", bidi works
> as expected.

Right.

> I believe this is due to the interaction of two things: 1. Emacs uses the first
> word of a paragraph to determine whether the paragraph is ltr or rtl; and 2.
> Emacs assumes that a paragraph is preceded by an empty line. So if you type rtl
> text on the line below the "--text follows this line--" separator, Emacs
> assumes the separator and the text you're typing belong to the same paragraph,
> and since the first word of this paragraph is from an ltr writing system, the
> paragraph is displayed ltr.

Exactly.



^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Message Mode and bidi
  2024-02-19 22:06       ` Christopher Culver via Emacs development discussions.
  2024-02-20  0:46         ` Eric Abrahamsen
@ 2024-02-20  3:31         ` Eli Zaretskii
  1 sibling, 0 replies; 14+ messages in thread
From: Eli Zaretskii @ 2024-02-20  3:31 UTC (permalink / raw)
  To: Christopher Culver; +Cc: joostkremers, emacs-devel

> Cc: emacs-devel@gnu.org
> Date: Mon, 19 Feb 2024 23:06:43 +0100
> From:  Christopher Culver via "Emacs development discussions." <emacs-devel@gnu.org>
> 
> Joost Kremers <joostkremers@fastmail.fm> writes:
> > When you compose a new message, is there a line "--text follows this line--"
> > separating the headers and the message text? In my case, there is (I use mu4e)
> > and when I type Arabic text on the line below this text, I get the effect you
> > mention. If I leave an empty line after "--text follows this line--", bidi works
> > as expected.
> 
> Indeed, if I just go down one line and then begin typing, bidi works as
> expected. I am feeling very foolish that I did not even try this. Thank
> you for clearing up this problem, and for shedding light on how Emacs
> considers paragraphs.

No need to apologize.  The way Emacs handles paragraphs for the
purpose of bidirectional text is described in the Emacs user manual,
in the node "Bidirectional Editing".



^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Message Mode and bidi
  2024-02-20  0:46         ` Eric Abrahamsen
@ 2024-02-20  3:37           ` Eli Zaretskii
  2024-02-20  5:16             ` Eric Abrahamsen
  0 siblings, 1 reply; 14+ messages in thread
From: Eli Zaretskii @ 2024-02-20  3:37 UTC (permalink / raw)
  To: Eric Abrahamsen; +Cc: emacs-devel

> From: Eric Abrahamsen <eric@ericabrahamsen.net>
> Date: Mon, 19 Feb 2024 16:46:22 -0800
> 
> Christopher Culver via "Emacs development discussions."
> <emacs-devel@gnu.org> writes:
> 
> > Joost Kremers <joostkremers@fastmail.fm> writes:
> >> When you compose a new message, is there a line "--text follows this line--"
> >> separating the headers and the message text? In my case, there is (I use mu4e)
> >> and when I type Arabic text on the line below this text, I get the effect you
> >> mention. If I leave an empty line after "--text follows this line--", bidi works
> >> as expected.
> >
> > Indeed, if I just go down one line and then begin typing, bidi works as
> > expected. I am feeling very foolish that I did not even try this. Thank
> > you for clearing up this problem, and for shedding light on how Emacs
> > considers paragraphs.
> 
> This is a bit weird, because the value of `mail-header-separator'
> ("--text follows this line--") is added to both `paragraph-start' and
> `paragraph-separate') in `message-mode'. You'd think one of those would
> do it.

Emacs doesn't use paragraph-separate and paragraph-start to define
where a paragraph starts and ends, for the purposes of determining the
base directionality of a paragraph.  It uses separate variables for
that, see the node "Bidirectional Editing" in the Emacs user manual.
The reason for using separate variables is because several modes,
including (but not limited to) message-mode, set the former variables
to regexps that get in the way of bidi reordering, and could easily
produce wrong results on display.



^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Message Mode and bidi
  2024-02-20  3:37           ` Eli Zaretskii
@ 2024-02-20  5:16             ` Eric Abrahamsen
  2024-02-20 14:36               ` Eli Zaretskii
  0 siblings, 1 reply; 14+ messages in thread
From: Eric Abrahamsen @ 2024-02-20  5:16 UTC (permalink / raw)
  To: emacs-devel


Eli Zaretskii <eliz@gnu.org> writes:

>> From: Eric Abrahamsen <eric@ericabrahamsen.net>
>> Date: Mon, 19 Feb 2024 16:46:22 -0800
>> 
>> Christopher Culver via "Emacs development discussions."
>> <emacs-devel@gnu.org> writes:
>> 
>> > Joost Kremers <joostkremers@fastmail.fm> writes:
>> >> When you compose a new message, is there a line "--text follows this line--"
>> >> separating the headers and the message text? In my case, there is (I use mu4e)
>> >> and when I type Arabic text on the line below this text, I get the effect you
>> >> mention. If I leave an empty line after "--text follows this line--", bidi works
>> >> as expected.
>> >
>> > Indeed, if I just go down one line and then begin typing, bidi works as
>> > expected. I am feeling very foolish that I did not even try this. Thank
>> > you for clearing up this problem, and for shedding light on how Emacs
>> > considers paragraphs.
>> 
>> This is a bit weird, because the value of `mail-header-separator'
>> ("--text follows this line--") is added to both `paragraph-start' and
>> `paragraph-separate') in `message-mode'. You'd think one of those would
>> do it.
>
> Emacs doesn't use paragraph-separate and paragraph-start to define
> where a paragraph starts and ends, for the purposes of determining the
> base directionality of a paragraph.  It uses separate variables for
> that, see the node "Bidirectional Editing" in the Emacs user manual.
> The reason for using separate variables is because several modes,
> including (but not limited to) message-mode, set the former variables
> to regexps that get in the way of bidi reordering, and could easily
> produce wrong results on display.

Do you think we'd stand a chance of finding values for
bidi-paragraph-start|separate-re that would resolve this particular
issue? Some of the message headers should be individually considered as
paragraphs for the purpose of bidi text as well, but that seems
considerably more challenging.




^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Message Mode and bidi
  2024-02-20  5:16             ` Eric Abrahamsen
@ 2024-02-20 14:36               ` Eli Zaretskii
  2024-02-28 16:54                 ` Eric Abrahamsen
  0 siblings, 1 reply; 14+ messages in thread
From: Eli Zaretskii @ 2024-02-20 14:36 UTC (permalink / raw)
  To: Eric Abrahamsen; +Cc: emacs-devel

> From: Eric Abrahamsen <eric@ericabrahamsen.net>
> Date: Mon, 19 Feb 2024 21:16:51 -0800
> 
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Eric Abrahamsen <eric@ericabrahamsen.net>
> >> Date: Mon, 19 Feb 2024 16:46:22 -0800
> >> 
> >> Christopher Culver via "Emacs development discussions."
> >> <emacs-devel@gnu.org> writes:
> >> 
> >> > Joost Kremers <joostkremers@fastmail.fm> writes:
> >> >> When you compose a new message, is there a line "--text follows this line--"
> >> >> separating the headers and the message text? In my case, there is (I use mu4e)
> >> >> and when I type Arabic text on the line below this text, I get the effect you
> >> >> mention. If I leave an empty line after "--text follows this line--", bidi works
> >> >> as expected.
> >> >
> >> > Indeed, if I just go down one line and then begin typing, bidi works as
> >> > expected. I am feeling very foolish that I did not even try this. Thank
> >> > you for clearing up this problem, and for shedding light on how Emacs
> >> > considers paragraphs.
> >> 
> >> This is a bit weird, because the value of `mail-header-separator'
> >> ("--text follows this line--") is added to both `paragraph-start' and
> >> `paragraph-separate') in `message-mode'. You'd think one of those would
> >> do it.
> >
> > Emacs doesn't use paragraph-separate and paragraph-start to define
> > where a paragraph starts and ends, for the purposes of determining the
> > base directionality of a paragraph.  It uses separate variables for
> > that, see the node "Bidirectional Editing" in the Emacs user manual.
> > The reason for using separate variables is because several modes,
> > including (but not limited to) message-mode, set the former variables
> > to regexps that get in the way of bidi reordering, and could easily
> > produce wrong results on display.
> 
> Do you think we'd stand a chance of finding values for
> bidi-paragraph-start|separate-re that would resolve this particular
> issue?

We already have those values described in the Emacs manual:

     Each paragraph of bidirectional text can have its own “base
  direction”, either right-to-left or left-to-right.  Text in
  left-to-right paragraphs begins on the screen at the left margin of the
  window and is truncated or continued when it reaches the right margin.
  By contrast, text in right-to-left paragraphs is displayed starting at
  the right margin and is continued or truncated at the left margin.  By
  default, paragraph boundaries are empty lines, i.e., lines consisting
  entirely of whitespace characters.  To change that, you can customize
  the two variables ‘bidi-paragraph-start-re’ and
  ‘bidi-paragraph-separate-re’, whose values should be regular expressions
  (strings); e.g., to have a single newline start a new paragraph, set
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  both of these variables to ‘"^"’.  These two variables are buffer-local
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  (*note Locals::).

But beware: doing this in Emacs will cause effects that are unpleasant
to readers of bidirectional text, because you could have "chess-like"
text display, like this:

asasasasasasasasasasasasassa
                                                ASASASASASASASASASASASASA
xcxcxcxcxcxcxcxcxcxcxcxcxc
                                                       JKJNJKKKNKNKNKNKNK

etc.  Here upper-case letters stand for RTL (like Arabic or Farsi)
text and lower-case letters stand for LTR (like Latin or Cyrillic)
text.

Let me explain.  UBA, the Unicode Bidirectional Algorithm which Emacs
implements, was designed for text-processing programs that wrap long
lines without inserting hard newlines.  For those applications, a
single hard newline indicates a new paragraph, and thus recalculating
the base direction of a paragraph after a newline is justified.

But in Emacs, we have a lot of text filled and wrapped using hard
newlines, and filling can insert a newline in an arbitrary place
inside the paragraph.  If it happens that a newline was inserted
before a strong right-to-left character, under the strict UBA rules
the next line will be considered as a paragraph with right-to-left
base direction, and rendered starting at the right window edge.  Which
leads to the above "chess-like" display.  Since that is basically
unacceptable for Emacs users, we require at least one empty line to
separate paragraphs.  The price of a single empty line before the body
of an email message that needs to be rendered right to left is a small
price to pay for solving the horrible display effect of changing the
paragraph's base direction after each newline.  An alternative would
be to use visual-line-mode instead of wrapping using hard newlines,
but that is still relatively rare in Emacs, especially in email
messages.



^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Message Mode and bidi
  2024-02-20 14:36               ` Eli Zaretskii
@ 2024-02-28 16:54                 ` Eric Abrahamsen
  2024-02-28 17:23                   ` Eli Zaretskii
  0 siblings, 1 reply; 14+ messages in thread
From: Eric Abrahamsen @ 2024-02-28 16:54 UTC (permalink / raw)
  To: emacs-devel

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Eric Abrahamsen <eric@ericabrahamsen.net>
>> Date: Mon, 19 Feb 2024 21:16:51 -0800
>> 
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> From: Eric Abrahamsen <eric@ericabrahamsen.net>
>> >> Date: Mon, 19 Feb 2024 16:46:22 -0800
>> >> 
>> >> Christopher Culver via "Emacs development discussions."
>> >> <emacs-devel@gnu.org> writes:
>> >> 
>> >> > Joost Kremers <joostkremers@fastmail.fm> writes:
>> >> >> When you compose a new message, is there a line "--text follows this line--"
>> >> >> separating the headers and the message text? In my case, there is (I use mu4e)
>> >> >> and when I type Arabic text on the line below this text, I get the effect you
>> >> >> mention. If I leave an empty line after "--text follows this line--", bidi works
>> >> >> as expected.
>> >> >
>> >> > Indeed, if I just go down one line and then begin typing, bidi works as
>> >> > expected. I am feeling very foolish that I did not even try this. Thank
>> >> > you for clearing up this problem, and for shedding light on how Emacs
>> >> > considers paragraphs.
>> >> 
>> >> This is a bit weird, because the value of `mail-header-separator'
>> >> ("--text follows this line--") is added to both `paragraph-start' and
>> >> `paragraph-separate') in `message-mode'. You'd think one of those would
>> >> do it.
>> >
>> > Emacs doesn't use paragraph-separate and paragraph-start to define
>> > where a paragraph starts and ends, for the purposes of determining the
>> > base directionality of a paragraph.  It uses separate variables for
>> > that, see the node "Bidirectional Editing" in the Emacs user manual.
>> > The reason for using separate variables is because several modes,
>> > including (but not limited to) message-mode, set the former variables
>> > to regexps that get in the way of bidi reordering, and could easily
>> > produce wrong results on display.
>> 
>> Do you think we'd stand a chance of finding values for
>> bidi-paragraph-start|separate-re that would resolve this particular
>> issue?
>
> We already have those values described in the Emacs manual:
>
>      Each paragraph of bidirectional text can have its own “base
>   direction”, either right-to-left or left-to-right.  Text in
>   left-to-right paragraphs begins on the screen at the left margin of the
>   window and is truncated or continued when it reaches the right margin.
>   By contrast, text in right-to-left paragraphs is displayed starting at
>   the right margin and is continued or truncated at the left margin.  By
>   default, paragraph boundaries are empty lines, i.e., lines consisting
>   entirely of whitespace characters.  To change that, you can customize
>   the two variables ‘bidi-paragraph-start-re’ and
>   ‘bidi-paragraph-separate-re’, whose values should be regular expressions
>   (strings); e.g., to have a single newline start a new paragraph, set
>                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   both of these variables to ‘"^"’.  These two variables are buffer-local
>   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>   (*note Locals::).
> 
> But beware: doing this in Emacs will cause effects that are unpleasant
> to readers of bidirectional text, because you could have "chess-like"
> text display, like this:
>
> asasasasasasasasasasasasassa
>                                                 ASASASASASASASASASASASASA
> xcxcxcxcxcxcxcxcxcxcxcxcxc
>                                                        JKJNJKKKNKNKNKNKNK
>
> etc.  Here upper-case letters stand for RTL (like Arabic or Farsi)
> text and lower-case letters stand for LTR (like Latin or Cyrillic)
> text.

This is the effect I was imagining in the header section of the message
buffer, and why I thought we should probably skip trying to handle this.
message-mode headers can be "continuation headers", as well, effectively
line-wrapped, which would make it even harder.

> Let me explain.  UBA, the Unicode Bidirectional Algorithm which Emacs
> implements, was designed for text-processing programs that wrap long
> lines without inserting hard newlines.  For those applications, a
> single hard newline indicates a new paragraph, and thus recalculating
> the base direction of a paragraph after a newline is justified.
>
> But in Emacs, we have a lot of text filled and wrapped using hard
> newlines, and filling can insert a newline in an arbitrary place
> inside the paragraph.  If it happens that a newline was inserted
> before a strong right-to-left character, under the strict UBA rules
> the next line will be considered as a paragraph with right-to-left
> base direction, and rendered starting at the right window edge.  Which
> leads to the above "chess-like" display.  Since that is basically
> unacceptable for Emacs users, we require at least one empty line to
> separate paragraphs.  The price of a single empty line before the body
> of an email message that needs to be rendered right to left is a small
> price to pay for solving the horrible display effect of changing the
> paragraph's base direction after each newline.  An alternative would
> be to use visual-line-mode instead of wrapping using hard newlines,
> but that is still relatively rare in Emacs, especially in email
> messages.

Thanks for the background! I guess I was hoping that we could at least
support OP's original request, which is making the first paragraph of
the message body independent of the mail header separator as regards
BIDI display. I experimented with putting the value of
`mail-header-separator' into `bidi-paragraph-start|separate-re', but
couldn't get it to display that first paragraph starting on the right.
Do you think this is feasible, and worth the effort?




^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Message Mode and bidi
  2024-02-28 16:54                 ` Eric Abrahamsen
@ 2024-02-28 17:23                   ` Eli Zaretskii
  2024-02-29  3:16                     ` Eric Abrahamsen
  0 siblings, 1 reply; 14+ messages in thread
From: Eli Zaretskii @ 2024-02-28 17:23 UTC (permalink / raw)
  To: Eric Abrahamsen; +Cc: emacs-devel

> From: Eric Abrahamsen <eric@ericabrahamsen.net>
> Date: Wed, 28 Feb 2024 08:54:54 -0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > But beware: doing this in Emacs will cause effects that are unpleasant
> > to readers of bidirectional text, because you could have "chess-like"
> > text display, like this:
> >
> > asasasasasasasasasasasasassa
> >                                                 ASASASASASASASASASASASASA
> > xcxcxcxcxcxcxcxcxcxcxcxcxc
> >                                                        JKJNJKKKNKNKNKNKNK
> >
> > etc.  Here upper-case letters stand for RTL (like Arabic or Farsi)
> > text and lower-case letters stand for LTR (like Latin or Cyrillic)
> > text.
> 
> This is the effect I was imagining in the header section of the message
> buffer, and why I thought we should probably skip trying to handle this.
> message-mode headers can be "continuation headers", as well, effectively
> line-wrapped, which would make it even harder.

Yes.  But the same can happen in the body of the message, and will IMO
be even more annoying, because people do care about the body.

> Thanks for the background! I guess I was hoping that we could at least
> support OP's original request, which is making the first paragraph of
> the message body independent of the mail header separator as regards
> BIDI display. I experimented with putting the value of
> `mail-header-separator' into `bidi-paragraph-start|separate-re', but
> couldn't get it to display that first paragraph starting on the right.
> Do you think this is feasible, and worth the effort?

No, I don't think it's worth the effort, even if you succeed.  The
price of leaving an empty line after the headers is so small that it
isn't worth the hassle to try to avoid it.  Once the user does that a
few times, the technique will be burned into his/her muscle memory (I
know because I went through that process myself, long ago).



^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: Message Mode and bidi
  2024-02-28 17:23                   ` Eli Zaretskii
@ 2024-02-29  3:16                     ` Eric Abrahamsen
  0 siblings, 0 replies; 14+ messages in thread
From: Eric Abrahamsen @ 2024-02-29  3:16 UTC (permalink / raw)
  To: Eli Zaretskii; +Cc: emacs-devel

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Eric Abrahamsen <eric@ericabrahamsen.net>
>> Date: Wed, 28 Feb 2024 08:54:54 -0800
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > But beware: doing this in Emacs will cause effects that are unpleasant
>> > to readers of bidirectional text, because you could have "chess-like"
>> > text display, like this:
>> >
>> > asasasasasasasasasasasasassa
>> >                                                 ASASASASASASASASASASASASA
>> > xcxcxcxcxcxcxcxcxcxcxcxcxc
>> >                                                        JKJNJKKKNKNKNKNKNK
>> >
>> > etc.  Here upper-case letters stand for RTL (like Arabic or Farsi)
>> > text and lower-case letters stand for LTR (like Latin or Cyrillic)
>> > text.
>> 
>> This is the effect I was imagining in the header section of the message
>> buffer, and why I thought we should probably skip trying to handle this.
>> message-mode headers can be "continuation headers", as well, effectively
>> line-wrapped, which would make it even harder.
>
> Yes.  But the same can happen in the body of the message, and will IMO
> be even more annoying, because people do care about the body.
>
>> Thanks for the background! I guess I was hoping that we could at least
>> support OP's original request, which is making the first paragraph of
>> the message body independent of the mail header separator as regards
>> BIDI display. I experimented with putting the value of
>> `mail-header-separator' into `bidi-paragraph-start|separate-re', but
>> couldn't get it to display that first paragraph starting on the right.
>> Do you think this is feasible, and worth the effort?
>
> No, I don't think it's worth the effort, even if you succeed.  The
> price of leaving an empty line after the headers is so small that it
> isn't worth the hassle to try to avoid it.  Once the user does that a
> few times, the technique will be burned into his/her muscle memory (I
> know because I went through that process myself, long ago).

Okay, fair enough! Thanks again.



^ permalink raw reply	[flat|nested] 14+ messages in thread

end of thread, other threads:[~2024-02-29  3:16 UTC | newest]

Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-02-19  1:12 Message Mode and bidi Christopher Culver via Emacs development discussions.
2024-02-19  3:36 ` Eli Zaretskii
2024-02-19 21:29   ` Christopher Culver via Emacs development discussions.
2024-02-19 21:50     ` Joost Kremers
2024-02-19 22:06       ` Christopher Culver via Emacs development discussions.
2024-02-20  0:46         ` Eric Abrahamsen
2024-02-20  3:37           ` Eli Zaretskii
2024-02-20  5:16             ` Eric Abrahamsen
2024-02-20 14:36               ` Eli Zaretskii
2024-02-28 16:54                 ` Eric Abrahamsen
2024-02-28 17:23                   ` Eli Zaretskii
2024-02-29  3:16                     ` Eric Abrahamsen
2024-02-20  3:31         ` Eli Zaretskii
2024-02-20  3:29       ` Eli Zaretskii

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/emacs.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).