bidi-display-reordering is now non-nil by default

bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

FYI

(Running for cover...)

Re: bidi-display-reordering is now non-nil by default

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

> FYI
>
> (Running for cover...)

I was expecting
(...revoc rof gninnuR)

-- 
David Kastrup

Re: bidi-display-reordering is now non-nil by default

[Juanma Barranquero]

On Thu, Jul 28, 2011 at 20:51, David Kastrup <dak@gnu.org> wrote:

> I was expecting
> (...revoc rof gninnuR)

That'd be a bug.

    Juanma

Re: bidi-display-reordering is now non-nil by default

[Werner LEMBERG]

> FYI
> 
> (Running for cover...)

I've just compiled Emacs from the bzr sources (revision 105365), using
mew as the mailing program.  Starting with this new default, scrolling
in mew's trash mail summary buffer (which contains a lot of Arabic
spam; in total, it has about 18000 lines currently) works extremely
bad -- this is on my GNU/Linux box with an i686 chip yielding 3325
BogoMIPS.  I'm very sorry, but for me the current behaviour is
completely unacceptable.

With bidi-dislay-reordering set to `nil', holding the the up or down
cursor keys pressed causes a smooth scrolling, either forwards or
backwards.  Setting it to `t', there is no smooth scrolling any more.
Instead, the cursor jumps, with large delays, to its final position.


    Werner

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> Date: Sun, 31 Jul 2011 00:55:16 +0200 (CEST)
> Cc: emacs-devel@gnu.org
> From: Werner LEMBERG <wl@gnu.org>
> 
> With bidi-dislay-reordering set to `nil', holding the the up or down
> cursor keys pressed causes a smooth scrolling, either forwards or
> backwards.  Setting it to `t', there is no smooth scrolling any more.
> Instead, the cursor jumps, with large delays, to its final position.

Does it help to set redisplay-dont-pause non-nil?

Re: bidi-display-reordering is now non-nil by default

[Werner LEMBERG]

>> With bidi-dislay-reordering set to `nil', holding the the up or
>> down cursor keys pressed causes a smooth scrolling, either forwards
>> or backwards.  Setting it to `t', there is no smooth scrolling any
>> more.  Instead, the cursor jumps, with large delays, to its final
>> position.
> 
> Does it help to set redisplay-dont-pause non-nil?

It helps a little bit, but it is still not responsive enough.  In
particular, since I've set the keyboard's character repeat rate to a
high value (approx. 30cps), I no longer see a moving cursor.

It seems to me that the longer the buffer, the worse it gets.  Is this
observation correct?

BTW, while being in a buffer, is there a quick way to change the value
of bidi-display-reordering?  Using set-variable doesn't work; it only
knows bidi-paragraph-direction.


   Werner

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> Date: Sun, 31 Jul 2011 08:21:22 +0200 (CEST)
> Cc: emacs-devel@gnu.org
> From: Werner LEMBERG <wl@gnu.org>
> 
> It seems to me that the longer the buffer, the worse it gets.  Is this
> observation correct?

Yes, see my other message in this thread.  Emacs is searching for the
beginning of the paragraph, but this whole buffer is one huge
paragraph.  The manual explains the significance of the paragraph
direction for the bidirectional display, in case you are interested in
the details.

> BTW, while being in a buffer, is there a quick way to change the value
> of bidi-display-reordering?

M-: (setq bidi-display-reordering nil) RET

After the first time, "M-: M-p" will avoid typing the whole stuff
again.

> Using set-variable doesn't work

This is on purpose: I don't think this should be a user-level
variable.  (Of course, for this to make sense, the display should be
reasonably responsive, and it is my intention to make it so, given
that the problems will be reported.)

Re: bidi-display-reordering is now non-nil by default

[Werner LEMBERG]

>> It seems to me that the longer the buffer, the worse it gets.  Is
>> this observation correct?
> 
> Yes, see my other message in this thread.  Emacs is searching for
> the beginning of the paragraph, but this whole buffer is one huge
> paragraph.  The manual explains the significance of the paragraph
> direction for the bidirectional display, in case you are interested
> in the details.

Thanks.  Does a limit exist (say, 100kByte) as a maximum value for a
paragraph length after that bidi-paragraph-direction is automatically
set to right-to-left (if not overridden specifically) to circumvent
this problem?

> This is on purpose: I don't think this should be a user-level
> variable.  (Of course, for this to make sense, the display should be
> reasonably responsive, and it is my intention to make it so, given
> that the problems will be reported.)

Thanks for your work.


    Werner

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> Date: Sun, 31 Jul 2011 08:44:35 +0200 (CEST)
> Cc: emacs-devel@gnu.org
> From: Werner LEMBERG <wl@gnu.org>
> 
> > Emacs is searching for the beginning of the paragraph, but this
> > whole buffer is one huge paragraph.  The manual explains the
> > significance of the paragraph direction for the bidirectional
> > display, in case you are interested in the details.
> 
> Thanks.  Does a limit exist (say, 100kByte) as a maximum value for a
> paragraph length after that bidi-paragraph-direction is automatically
> set to right-to-left (if not overridden specifically) to circumvent
> this problem?

You mean left-to-right, not right-to-left, yes?  Setting the base
paragraph direction in that buffer to right-to-left will make the
display barely legible.

There's no such feature currently, no.  It could be easily added, but
I'm not sure such a heuristics is justified; it certainly can Do The
Wrong Thing in some cases, e.g., I could imagine a similar use case
with some mail summary buffer, but with the message numbers spelled in
Arabic digits and most of the Subject lines in Arabic or Farsi.  In
that case, the correct direction is actually right-to-left.  As
another example, I have a large text file with the entire Bible in
Hebrew, which is again one huge paragraph -- but its direction should
be right-to-left.

The design of determining the base paragraph direction dynamically as
part of redisplay is based on the assumption that most buffers either
have small enough paragraphs to not cause any significant slowdown, or
else are specialized buffers whose correct base direction is 100%
predictable in advance.  If you (or someone else) can see use cases
that break this assumption, please describe those use cases.

Re: bidi-display-reordering is now non-nil by default

[Werner LEMBERG]

>> Thanks.  Does a limit exist (say, 100kByte) as a maximum value for
>> a paragraph length after that bidi-paragraph-direction is
>> automatically set to right-to-left (if not overridden specifically)
>> to circumvent this problem?
> 
> You mean left-to-right, not right-to-left, yes?

Yes.  Sorry for the typo.

> There's no such feature currently, no.  It could be easily added,
> but I'm not sure such a heuristics is justified; it certainly can Do
> The Wrong Thing in some cases, e.g., I could imagine a similar use
> case with some mail summary buffer, but with the message numbers
> spelled in Arabic digits and most of the Subject lines in Arabic or
> Farsi.  In that case, the correct direction is actually
> right-to-left.  As another example, I have a large text file with
> the entire Bible in Hebrew, which is again one huge paragraph -- but
> its direction should be right-to-left.

Yes, such a limit would do the wrong thing in many cases, but we are
talking about a technical limitation: the question is whether it is
more important to have acceptable cursor motion with in a buffer or
correct display.  I could even imagine that emacs implements a
function to time cursor movements at various places within the buffer.
If it takes too long, emacs should do something, either modifying the
value of bidi-paragraph-direction, or artificially inserting a
paragraph separator, or whatever.


    Werner

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> Date: Sun, 31 Jul 2011 00:55:16 +0200 (CEST)
> Cc: emacs-devel@gnu.org
> From: Werner LEMBERG <wl@gnu.org>
> 
> I've just compiled Emacs from the bzr sources (revision 105365), using
> mew as the mailing program.  Starting with this new default, scrolling
> in mew's trash mail summary buffer (which contains a lot of Arabic
> spam; in total, it has about 18000 lines currently) works extremely
> bad -- this is on my GNU/Linux box with an i686 chip yielding 3325
> BogoMIPS.  I'm very sorry, but for me the current behaviour is
> completely unacceptable.
> 
> With bidi-dislay-reordering set to `nil', holding the the up or down
> cursor keys pressed causes a smooth scrolling, either forwards or
> backwards.  Setting it to `t', there is no smooth scrolling any more.
> Instead, the cursor jumps, with large delays, to its final position.

If you set bidi-paragraph-direction to 'left-to-right in that buffer,
does the problem go away?

Re: bidi-display-reordering is now non-nil by default

[Werner LEMBERG]

>> With bidi-dislay-reordering set to `nil', holding the the up or
>> down cursor keys pressed causes a smooth scrolling, either forwards
>> or backwards.  Setting it to `t', there is no smooth scrolling any
>> more.  Instead, the cursor jumps, with large delays, to its final
>> position.
> 
> If you set bidi-paragraph-direction to 'left-to-right in that
> buffer, does the problem go away?

Yes.


    Werner

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> Date: Sun, 31 Jul 2011 08:27:21 +0200 (CEST)
> Cc: emacs-devel@gnu.org
> From: Werner LEMBERG <wl@gnu.org>
> 
> >> With bidi-dislay-reordering set to `nil', holding the the up or
> >> down cursor keys pressed causes a smooth scrolling, either forwards
> >> or backwards.  Setting it to `t', there is no smooth scrolling any
> >> more.  Instead, the cursor jumps, with large delays, to its final
> >> position.
> > 
> > If you set bidi-paragraph-direction to 'left-to-right in that
> > buffer, does the problem go away?
> 
> Yes.

Thanks.  So this is the solution for this buffer (and for any other
buffer whose paragraphs should always have the left-to-right base
direction, i.e. their lines should always start at the left margin of
the window).

As mew is not part of Emacs (I think), what would be the best way of
achieving this effect?  I'd prefer not to ask users to do that
manually, if that can be avoided.  Does this buffer have a fixed name
we can rely upon?

Chong, Stefan: how about having a (customizable) list of buffer names
which should by default have left-to-right paragraph direction?  Would
this be a reasonable solution?  We could add buffer names to this list
as we discover more such cases.

Re: bidi-display-reordering is now non-nil by default

[Werner LEMBERG]

[This is about behaviour of current bzr Emacs in really large
.mew-summary files.]

>> >> With bidi-dislay-reordering set to `nil', holding the the up or
>> >> down cursor keys pressed causes a smooth scrolling, either
>> >> forwards or backwards.  Setting it to `t', there is no smooth
>> >> scrolling any more.  Instead, the cursor jumps, with large
>> >> delays, to its final position.
>> > 
>> > If you set bidi-paragraph-direction to 'left-to-right in that
>> > buffer, does the problem go away?
>> 
>> Yes.
> 
> Thanks.  So this is the solution for this buffer (and for any other
> buffer whose paragraphs should always have the left-to-right base
> direction, i.e. their lines should always start at the left margin
> of the window).
> 
> As mew is not part of Emacs (I think), what would be the best way of
> achieving this effect?  I'd prefer not to ask users to do that
> manually, if that can be avoided.  Does this buffer have a fixed
> name we can rely upon?

Yes, the name is always `.mew-summary'.  I've CCed the mew mailing
list so that the author of mew gets informed.  It should be
straightforward to add a proper call to `bidi-paragraph-direction'
within the next mew release.

> Chong, Stefan: how about having a (customizable) list of buffer
> names which should by default have left-to-right paragraph
> direction?  Would this be a reasonable solution?  We could add
> buffer names to this list as we discover more such cases.


    Werner

Re: bidi-display-reordering is now non-nil by default

[David Kastrup]

Werner LEMBERG <wl@gnu.org> writes:

> [This is about behaviour of current bzr Emacs in really large
> .mew-summary files.]
>
>>> >> With bidi-dislay-reordering set to `nil', holding the the up or
>>> >> down cursor keys pressed causes a smooth scrolling, either
>>> >> forwards or backwards.  Setting it to `t', there is no smooth
>>> >> scrolling any more.  Instead, the cursor jumps, with large
>>> >> delays, to its final position.
>>> > 
>>> > If you set bidi-paragraph-direction to 'left-to-right in that
>>> > buffer, does the problem go away?
>>> 
>>> Yes.
>> 
>> Thanks.  So this is the solution for this buffer (and for any other
>> buffer whose paragraphs should always have the left-to-right base
>> direction, i.e. their lines should always start at the left margin
>> of the window).
>> 
>> As mew is not part of Emacs (I think), what would be the best way of
>> achieving this effect?  I'd prefer not to ask users to do that
>> manually, if that can be avoided.  Does this buffer have a fixed
>> name we can rely upon?
>
> Yes, the name is always `.mew-summary'.  I've CCed the mew mailing
> list so that the author of mew gets informed.  It should be
> straightforward to add a proper call to `bidi-paragraph-direction'
> within the next mew release.

I think that is a bottomless pit.  I consider it nonsensical to treat a
block of aligned lines as one large paragraph where one would need to
find out the paragraph direction of the whole.  It might be nice to have
the same default paragraph direction in the whole "paragraph" for
alignment purposes, but it is nonsensical to wait for the paragraph to
complete.

With a bit of luck, the alignment uses display properties (because
otherwise proportional fonts are raining on the parade), and in the
presence of a :space display property or what our aligning properties
were called, I think that the paragraph direction should be determined
at the end of the first line, rather than the whole "paragraph".

It might also be reasonable to set the paragraph boundary detection
regexps to nil in buffers where no paragraphs will ever be found, and
let bidi revert to a (locale-dependent?) default direction for such
buffers.

-- 
David Kastrup

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Sun, 31 Jul 2011 09:59:17 +0200
> Cc: mew-int@mew.org

I've removed the mew list from the CC, as this doesn't seem to be
related to mew.

> > Yes, the name is always `.mew-summary'.  I've CCed the mew mailing
> > list so that the author of mew gets informed.  It should be
> > straightforward to add a proper call to `bidi-paragraph-direction'
> > within the next mew release.
> 
> I think that is a bottomless pit.  I consider it nonsensical to treat a
> block of aligned lines as one large paragraph where one would need to
> find out the paragraph direction of the whole.

The paragraph direction is determined by the paragraph beginning, not
by all of it.  The first "strong directional" character in the
paragraph, either a left-to-right character or a right-to-left
character, will determine the base direction of that paragraph.  E.g.,
a single Latin letter is all that's needed for determining that the
paragraph direction is left-to-right.

> It might be nice to have the same default paragraph direction in the
> whole "paragraph" for alignment purposes

Alignment is not the issue here; correct display as expected by users
is.

> but it is nonsensical to wait for the paragraph to complete.

Emacs doesn't wait for the paragraph to complete, to determine its
base direction.  All it needs is its beginning; one character is
enough.

The use case that Werner reported was with point being near the end of
a very large buffer that doesn't have paragraph breaks.  When point
was at the beginning of that same large buffer, cursor motion was
responsive enough.  That's because, when redisplay is called in the
middle of a paragraph, and bidi-paragraph-direction is nil, Emacs
searches back to the beginning of the paragraph to determine its
direction, and in this case it would search back to the beginning of
this very large buffer.

> With a bit of luck, the alignment uses display properties (because
> otherwise proportional fonts are raining on the parade), and in the
> presence of a :space display property or what our aligning properties
> were called

Unless I misunderstand you, the display properties are irrelevant to
the issue at hand, as is alignment in general.

> I think that the paragraph direction should be determined
> at the end of the first line, rather than the whole "paragraph".

As explained above, it is determined at the first strong directional
character, much sooner than the end of the first line.

> It might also be reasonable to set the paragraph boundary detection
> regexps to nil in buffers where no paragraphs will ever be found

How to detect these buffers?  If we can detect them reliably, we could
probably always set their paragraph direction to left-to-right, which
would be a better default than a locale-dependent one (see below).

> and let bidi revert to a (locale-dependent?) default direction for
> such buffers.

Experience shows that some people who need the bidirectional display
work in locales that have nothing to do with R2L scripts.  So I think
it would be a bad misfeature to do what you suggest (IIUC).  At least
we should consider better alternatives before we give up in this way.

Re: bidi-display-reordering is now non-nil by default

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>> Date: Sun, 31 Jul 2011 09:59:17 +0200
>> Cc: mew-int@mew.org
>
> I've removed the mew list from the CC, as this doesn't seem to be
> related to mew.
>
>> > Yes, the name is always `.mew-summary'.  I've CCed the mew mailing
>> > list so that the author of mew gets informed.  It should be
>> > straightforward to add a proper call to `bidi-paragraph-direction'
>> > within the next mew release.
>> 
>> I think that is a bottomless pit.  I consider it nonsensical to treat a
>> block of aligned lines as one large paragraph where one would need to
>> find out the paragraph direction of the whole.
>
> The paragraph direction is determined by the paragraph beginning, not
> by all of it.  The first "strong directional" character in the
> paragraph, either a left-to-right character or a right-to-left
> character, will determine the base direction of that paragraph.  E.g.,
> a single Latin letter is all that's needed for determining that the
> paragraph direction is left-to-right.

So we are apparently searching for the paragraph beginning of a
humongous paragraph for each keystroke.

>> It might also be reasonable to set the paragraph boundary detection
>> regexps to nil in buffers where no paragraphs will ever be found
>
> How to detect these buffers?

I was not suggesting detecting them.  I was suggesting that modes
creating buffers where paragraphs are not a usable concept disable
paragraph detection.  Manually.  That's another bottomless pit, but at
least it approaches the problem from an angle that is not bidi-related
and might make sense on versions (including XEmacs) that don't bother
about bidi at all and consequently don't support
bidi-paragraph-direction.

-- 
David Kastrup

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Sun, 31 Jul 2011 11:15:24 +0200
> 
> > The paragraph direction is determined by the paragraph beginning, not
> > by all of it.  The first "strong directional" character in the
> > paragraph, either a left-to-right character or a right-to-left
> > character, will determine the base direction of that paragraph.  E.g.,
> > a single Latin letter is all that's needed for determining that the
> > paragraph direction is left-to-right.
> 
> So we are apparently searching for the paragraph beginning of a
> humongous paragraph for each keystroke.

In the case in point, yes.  Also for each cursor motion command.

> >> It might also be reasonable to set the paragraph boundary detection
> >> regexps to nil in buffers where no paragraphs will ever be found
> >
> > How to detect these buffers?
> 
> I was not suggesting detecting them.  I was suggesting that modes
> creating buffers where paragraphs are not a usable concept disable
> paragraph detection.  Manually.

Setting bidi-paragraph-direction to a non-nil value has precisely the
same effect, except that it will also DTRT when the mode wants all its
paragraphs have right-to-left direction.  Disabling detection of
paragraph direction can only support left-to-right, by contrast (or
some other static default).

> That's another bottomless pit

We could do this through the modes (buffer-menu.el already does), but
my concern was about modes that are not bundled with Emacs, such as
mew, which was what triggered this thread in the first place.  There's
also a bug #9209 about a similar problem with vm, another unbundled
package; I haven't yet heard from the OP in that case, but I presume
the reason is identical.

That is why I suggested to have a list of known names of buffers that
need a certain fixed value of bidi-paragraph-direction.  But that idea
is by no means sacred; if there are better ones, let's hear them.

> but at least it approaches the problem from an angle that is not
> bidi-related and might make sense on versions (including XEmacs)
> that don't bother about bidi at all and consequently don't support
> bidi-paragraph-direction.

bidi-paragraph-direction is a variable.  Setting a variable cannot
hurt, even if the version of Emacs one runs doesn't know about it.  Am
I missing something?

Re: bidi-display-reordering is now non-nil by default

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>> Date: Sun, 31 Jul 2011 11:15:24 +0200
>> 
>> I was not suggesting detecting them.  I was suggesting that modes
>> creating buffers where paragraphs are not a usable concept disable
>> paragraph detection.  Manually.
>
> Setting bidi-paragraph-direction to a non-nil value has precisely the
> same effect, except that it will also DTRT when the mode wants all its
> paragraphs have right-to-left direction.  Disabling detection of
> paragraph direction can only support left-to-right, by contrast (or
> some other static default).
>
>> That's another bottomless pit
>
> We could do this through the modes (buffer-menu.el already does), but
> my concern was about modes that are not bundled with Emacs, such as
> mew, which was what triggered this thread in the first place.

How about setting bidi-paragraph-direction to nil only in text-mode,
giving it a non-nil default otherwise?

That would seem to have a much better hit/miss ratio as specializing on
buffer names.

>> but at least it approaches the problem from an angle that is not
>> bidi-related and might make sense on versions (including XEmacs) that
>> don't bother about bidi at all and consequently don't support
>> bidi-paragraph-direction.
>
> bidi-paragraph-direction is a variable.  Setting a variable cannot
> hurt, even if the version of Emacs one runs doesn't know about it.  Am
> I missing something?

The willingness of maintainers to add code for features that are
irrelevant to the functionality of their package and their Emacs
version?

"My mode does not have bidi, so why should I tell Emacs it has a bidi
direction when it makes no user-visible difference except avoiding a
performance bug" is harder to convincingly counter than "My mode does
not have paragraphs, so why should I tell Emacs it does not have
paragraphs when it makes a user-visible difference with the paragraph
movement commands".

-- 
David Kastrup

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Sun, 31 Jul 2011 11:54:19 +0200
> 
> > We could do this through the modes (buffer-menu.el already does), but
> > my concern was about modes that are not bundled with Emacs, such as
> > mew, which was what triggered this thread in the first place.
> 
> How about setting bidi-paragraph-direction to nil only in text-mode,
> giving it a non-nil default otherwise?

I actually thought about doing that, but the disadvantage is that some
popular modes where dynamic determination of paragraph direction is
important do not inherit from Text Mode.  Rmail Mode is one example; I
expect Gnus to have similar issues.  IOW, any buffer where email is
displayed for reading would probably be affected.

That's just one consideration that prevented me from doing what you
suggest.  I expect other mode families to exist that have the same
issue.

But if we can cover them somehow, that would be a good solution, I
think.

> > bidi-paragraph-direction is a variable.  Setting a variable cannot
> > hurt, even if the version of Emacs one runs doesn't know about it.  Am
> > I missing something?
> 
> The willingness of maintainers to add code for features that are
> irrelevant to the functionality of their package and their Emacs
> version?

Well, I thought compatibility to Emacs in general and newer versions
in particular should be important, especially if the price is a single
line of Lisp.  But I have no experience in dealing with package
maintainers.

> "My mode does not have bidi, so why should I tell Emacs it has a bidi
> direction when it makes no user-visible difference except avoiding a
> performance bug" is harder to convincingly counter than "My mode does
> not have paragraphs, so why should I tell Emacs it does not have
> paragraphs when it makes a user-visible difference with the paragraph
> movement commands".

Maybe.  Bidi is a feature of display, i.e. it's impossible to
correctly display text in some scripts without it.  With today's deep
and wide m17n, whereby file names and network addresses can use R2L
characters, saying "my mode doesn't have bidi" is tantamount to saying
"my mode doesn't support certain cultures".  Whether this is
reasonable or not, I cannot judge.

Re: bidi-display-reordering is now non-nil by default

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

> Maybe.  Bidi is a feature of display, i.e. it's impossible to
> correctly display text in some scripts without it.  With today's deep
> and wide m17n, whereby file names and network addresses can use R2L
> characters, saying "my mode doesn't have bidi" is tantamount to saying
> "my mode doesn't support certain cultures".

Emacs is a desktop environment with at best marginal
internationalization, and editing texts is only a small part of its
functionality.  I think it is delusional to prescribe that all modes and
programming languages should in the name of culture-friendliness be
thought of as bidirectional by default when virtually no aspect of
Emacs' user interfaces (short of tutorials in multiple languages)
departs from English.

Making Emacs viable for working with bidirectional material is an
excellent thing.  But Emacs needs to make much greater strides in
internationalization before it makes sense to have bidi efforts be
opt-out rather than opt-in in modes and buffers.  There is a reason
text-mode is opt-out rather than opt-in, and I think it would cause much
less trouble if we make bidi-paragraph-direction default to
left-to-right, set it to nil in text-mode, and add similar settings to
RMAIL and other modes on an as-needed basis.

> Whether this is reasonable or not, I cannot judge.

Everything can be reasoned, but we still need to make choices.

-- 
David Kastrup

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Sun, 31 Jul 2011 12:35:40 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Maybe.  Bidi is a feature of display, i.e. it's impossible to
> > correctly display text in some scripts without it.  With today's deep
> > and wide m17n, whereby file names and network addresses can use R2L
> > characters, saying "my mode doesn't have bidi" is tantamount to saying
> > "my mode doesn't support certain cultures".
> 
> Emacs is a desktop environment with at best marginal
> internationalization, and editing texts is only a small part of its
> functionality.  I think it is delusional to prescribe that all modes and
> programming languages should in the name of culture-friendliness be
> thought of as bidirectional by default when virtually no aspect of
> Emacs' user interfaces (short of tutorials in multiple languages)
> departs from English.

I wasn't talking about the UI, I was talking mainly about the text we
display in the window.  For example, Dired mode can say "I have no
bidi", but then it will display file names that use R2L characters in
a way that makes them barely readable.  Rmail can say "I have no
bidi", but then email messages written in R2L scripts will be
unreadable.  Etc., etc.  This happens because R2L scripts from outside
Emacs seep into Emacs, whether we want that or not.

Btw, some small portion of this leaks into the UI as well.  For
example, the name of a buffer that visits a file whose name includes
R2L characters will be displayed in the mode line and in the
appropriate menu items.  (The former is supported by the current code
base, the latter depends on the toolkit, but should generally work as
well.)  So, while it's true that the Emacs is only marginally
internationalized, the environment in which we work every day already
requires that even the UI should support bidi.  Otherwise, users who
_really_ need bidi will not be satisfied, and will turn to other
applications for their needs.  The net effect will be that the
addition of bidi to Emacs will become a futile exercise.

Re: bidi-display-reordering is now non-nil by default

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>> Date: Sun, 31 Jul 2011 12:35:40 +0200
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > Maybe.  Bidi is a feature of display, i.e. it's impossible to
>> > correctly display text in some scripts without it.  With today's deep
>> > and wide m17n, whereby file names and network addresses can use R2L
>> > characters, saying "my mode doesn't have bidi" is tantamount to saying
>> > "my mode doesn't support certain cultures".
>> 
>> Emacs is a desktop environment with at best marginal
>> internationalization, and editing texts is only a small part of its
>> functionality.  I think it is delusional to prescribe that all modes and
>> programming languages should in the name of culture-friendliness be
>> thought of as bidirectional by default when virtually no aspect of
>> Emacs' user interfaces (short of tutorials in multiple languages)
>> departs from English.
>
> I wasn't talking about the UI, I was talking mainly about the text we
> display in the window.  For example, Dired mode can say "I have no
> bidi", but then it will display file names that use R2L characters in
> a way that makes them barely readable.

Because of a setting of bidi-paragraph-direction to left-to-right?  I
should be surprised.

> Rmail can say "I have no bidi", but then email messages written in R2L
> scripts will be unreadable.  Etc., etc.  This happens because R2L
> scripts from outside Emacs seep into Emacs, whether we want that or
> not.

Aren't you confusing the arguments for the setting of
bidi-paragraph-direction with that of bidi-display-reordering?

> Btw, some small portion of this leaks into the UI as well.  For
> example, the name of a buffer that visits a file whose name includes
> R2L characters will be displayed in the mode line and in the
> appropriate menu items.

And you would consider it a good idea if the mode line flipped its
orientation depending on the first directional letter in it?

> So, while it's true that the Emacs is only marginally
> internationalized, the environment in which we work every day already
> requires that even the UI should support bidi.  Otherwise, users who
> _really_ need bidi will not be satisfied, and will turn to other
> applications for their needs.  The net effect will be that the
> addition of bidi to Emacs will become a futile exercise.

I think you got your priorities mixed up.  There is a different between
supporting bidirectional text editing, and applauding side-effects of
the implementation showing up where they serve no purpose.

If you don't want to have people disable bidi-display-reordering as a
sanity measure (and .emacs cross-pollination is an annoying but rather
present fact of life), you better be conservative with making
bidi-paragraph-direction assume a default setting causing significant
regressions where they serve no purpose even for people using R2L
languages regularly.

-- 
David Kastrup

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Sun, 31 Jul 2011 15:23:13 +0200
> 
> > I wasn't talking about the UI, I was talking mainly about the text we
> > display in the window.  For example, Dired mode can say "I have no
> > bidi", but then it will display file names that use R2L characters in
> > a way that makes them barely readable.
> 
> Because of a setting of bidi-paragraph-direction to left-to-right?  I
> should be surprised.
> 
> > Rmail can say "I have no bidi", but then email messages written in R2L
> > scripts will be unreadable.  Etc., etc.  This happens because R2L
> > scripts from outside Emacs seep into Emacs, whether we want that or
> > not.
> 
> Aren't you confusing the arguments for the setting of
> bidi-paragraph-direction with that of bidi-display-reordering?

Maybe I was confused by your statement, which sounded much broader
than just the paragraph direction:

> Emacs is a desktop environment with at best marginal
> internationalization, and editing texts is only a small part of its
> functionality.  I think it is delusional to prescribe that all modes and
> programming languages should in the name of culture-friendliness be
> thought of as bidirectional by default when virtually no aspect of
> Emacs' user interfaces (short of tutorials in multiple languages)
> departs from English.

This talks about bidirectionality in general, not about paragraph
direction.  If that's not what you meant, apologies for my
misunderstanding.

> > Btw, some small portion of this leaks into the UI as well.  For
> > example, the name of a buffer that visits a file whose name includes
> > R2L characters will be displayed in the mode line and in the
> > appropriate menu items.
> 
> And you would consider it a good idea if the mode line flipped its
> orientation depending on the first directional letter in it?

Are you saying that Emacs behaves like this?  If not, then the code I
wrote already answers this question, doesn't it?

> If you don't want to have people disable bidi-display-reordering as a
> sanity measure (and .emacs cross-pollination is an annoying but rather
> present fact of life), you better be conservative with making
> bidi-paragraph-direction assume a default setting causing significant
> regressions where they serve no purpose even for people using R2L
> languages regularly.

There's no need to threaten.  This whole side track blew the issue out
of proportion.  For the record, I have no objection to changing the
default of bidi-paragraph-direction to left-to-right, and then setting
it nil selectively in some modes.  I also have no objection to leaving
it at nil, and setting it to something else in select modes or buffers
that go by certain names.  I'll let Stefan and Chong decide what is
best.

Re: bidi-display-reordering is now non-nil by default

[Lars Magne Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> So we are apparently searching for the paragraph beginning of a
>> humongous paragraph for each keystroke.
>
> In the case in point, yes.  Also for each cursor motion command.

Can't this information be cached somehow?

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Date: Sun, 31 Jul 2011 13:07:13 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> So we are apparently searching for the paragraph beginning of a
> >> humongous paragraph for each keystroke.
> >
> > In the case in point, yes.  Also for each cursor motion command.
> 
> Can't this information be cached somehow?

I think it would be a very complicated caching, because you cannot
determine whether you are still inside the same paragraph without
finding the beginning and the end of that paragraph.  Since Emacs is
an editor, i.e. most buffers get their text modified, you'd need to
update your notion of the paragraph beginning and end very frequently,
so you are back at the same problem: the need to find the paragraph
beginning as a prerequisite for correct redisplay (but now you also
need to find its end, which will eat up more cycles).

Even with paragraph beginning and end in hand, there are still
complications, because inserting a character at the paragraph
beginning can change its base direction.  And that character does not
have to be the very first character of the paragraph, it can be
preceded by any number of so-called "neutral" and "weak" characters,
like punctuation and whitespace, that have no directionality of their
own, but instead take the direction of surrounding "strong"
characters.

Doing this correctly and efficiently will require non-trivial
infrastructure, which is not something to consider when the pretest is
about to start, I think.

So I'd rather we didn't implement such a cache if it can be avoided.

Re: bidi-display-reordering is now non-nil by default

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

> Doing this correctly and efficiently will require non-trivial
> infrastructure, which is not something to consider when the pretest is
> about to start, I think.
>
> So I'd rather we didn't implement such a cache if it can be avoided.

If we don't have the infrastructure for supporting a default setting in
a way tolerable for most users, the default needs to be different until
we do.

-- 
David Kastrup

Re: bidi-display-reordering is now non-nil by default

[Lars Magne Ingebrigtsen]

David Kastrup <dak@gnu.org> writes:

> If we don't have the infrastructure for supporting a default setting in
> a way tolerable for most users, the default needs to be different until
> we do.

We need to get this stuff tested, so switching the default to non-nil is
the right decision.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: bidi-display-reordering is now non-nil by default

[David Kastrup]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> David Kastrup <dak@gnu.org> writes:
>
>> If we don't have the infrastructure for supporting a default setting in
>> a way tolerable for most users, the default needs to be different until
>> we do.
>
> We need to get this stuff tested, so switching the default to non-nil is
> the right decision.

I repeat: we are currently talking about the default of
bidi-paragraph-direction which Eli argues should be nil in all modes
(not just text-mode derived ones) that don't explicitly override it.

I consider that a bad idea _even_ assuming it did not cause major
performance regressions.

There are many modes in Emacs that are doing some sort of tabular
display, and flipping some vertical areas of the display horizontally
around depending on the directionality of the first explicit character
is going to make it _harder_, not easier for people trying to work with
bidi content to get sensible results.

So I repeat my contention that a general default of
bidi-paragraph-direction to nil is ill-advised, even discounting the
performance regression that will see no fix in the impending release.

I am not arguing for a default of bidi-display-reordering as nil, in
which case the default of bidi-paragraph-direction would not be a
concern for most users.

So please stop beating that straw man.  The current point of discussion
is bidi-paragraph-direction, not bidi-display-reordering.

-- 
David Kastrup

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Sun, 31 Jul 2011 15:54:01 +0200
> 
> I repeat: we are currently talking about the default of
> bidi-paragraph-direction which Eli argues should be nil in all modes
> (not just text-mode derived ones) that don't explicitly override it.

I never said that.  I just mentioned that as a possible solution.

> There are many modes in Emacs that are doing some sort of tabular
> display, and flipping some vertical areas of the display horizontally
> around depending on the directionality of the first explicit character
> is going to make it _harder_, not easier for people trying to work with
> bidi content to get sensible results.

Which modes are you talking about, specifically?

Re: bidi-display-reordering is now non-nil by default

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>> Date: Sun, 31 Jul 2011 15:54:01 +0200
>> 
>> I repeat: we are currently talking about the default of
>> bidi-paragraph-direction which Eli argues should be nil in all modes
>> (not just text-mode derived ones) that don't explicitly override it.
>
> I never said that.  I just mentioned that as a possible solution.

There were explicit plans discussed for making a buffer-name list where
a nil default would get overriden.  I consider that approach a
non-starter.  Or rather a non-finisher.

>> There are many modes in Emacs that are doing some sort of tabular
>> display, and flipping some vertical areas of the display horizontally
>> around depending on the directionality of the first explicit character
>> is going to make it _harder_, not easier for people trying to work with
>> bidi content to get sensible results.
>
> Which modes are you talking about, specifically?

dired, buffer-list, ses, lots of things.  Basically almost everything
that one likes using fixed-width fonts for (or which uses alignment
properties in order to still look good with variable-pitch fonts) would
likely get awkward with nil bidi-paragraph-direction.

-- 
David Kastrup

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Sun, 31 Jul 2011 16:26:12 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: David Kastrup <dak@gnu.org>
> >> Date: Sun, 31 Jul 2011 15:54:01 +0200
> >> 
> >> I repeat: we are currently talking about the default of
> >> bidi-paragraph-direction which Eli argues should be nil in all modes
> >> (not just text-mode derived ones) that don't explicitly override it.
> >
> > I never said that.  I just mentioned that as a possible solution.
> 
> There were explicit plans discussed for making a buffer-name list where
> a nil default would get overriden.

That was one idea, yes.  It still is, and IMO should be considered.

> I consider that approach a non-starter.  Or rather a non-finisher.

Look, you only learned about this issue this morning.  I don't think
it's wise to make up our minds about the best solution after so little
thought.  I think we need more data points and more rational
discussions of the various aspects of this.  I would also like to hear
at least one other serious user of R2L scripts.  This is an important
usability issue, it shouldn't be decided on a knee-jerk basis.

> >> There are many modes in Emacs that are doing some sort of tabular
> >> display, and flipping some vertical areas of the display horizontally
> >> around depending on the directionality of the first explicit character
> >> is going to make it _harder_, not easier for people trying to work with
> >> bidi content to get sensible results.
> >
> > Which modes are you talking about, specifically?
> 
> dired, buffer-list, ses

Dired is not affected, as the first character is always strong L.
Buffer-list already sets bidi-paragraph-direction to left-to-right.
As for SES Mode, I couldn't get it to display R2L characters, probably
because I don't know anything about SES, and couldn't find any docs.
(It looks like it uses symbols instead of strings?)  If someone can
show a recipe for doing that, I will look into it.

> lots of things.  Basically almost everything
> that one likes using fixed-width fonts for (or which uses alignment
> properties in order to still look good with variable-pitch fonts) would
> likely get awkward with nil bidi-paragraph-direction.

I think we should raise and discuss specific examples.  There's no
ideology here, just pragmatic choices, so we should study several data
points to see which alternative is better.

Re: bidi-display-reordering is now non-nil by default

[Mohsen BANAN]

>>>>> On Sun, 31 Jul 2011 18:44:03 +0300, Eli Zaretskii <eliz@gnu.org> said:

  >> From: David Kastrup <dak@gnu.org>
  >> Date: Sun, 31 Jul 2011 16:26:12 +0200

  David> I consider that approach a non-starter.  Or rather a non-finisher.

  Eli> Look, you only learned about this issue this morning.  I don't think
  Eli> it's wise to make up our minds about the best solution after so little
  Eli> thought.  I think we need more data points and more rational
  Eli> discussions of the various aspects of this.  I would also like to hear
  Eli> at least one other serious user of R2L scripts.  This is an important
  Eli> usability issue, it shouldn't be decided on a knee-jerk basis.

I agree. 

This is a good point to spend some time and
formulate some goals, policies and strategies.

I consider myself a serious user of R2L scripts
(Persian/Farsi). And here is my $0.02.

With bidi-display-reordering now set to non-nil by
default, a new chapter has been opened.

The previous chapter was adding bidi to emacs as
an editor.

The current chapter is that of adding bidi to
emacs as a User Environment (even broader than a
desktop environment).

That will likely involve modifications to many
emacs packages.

Prior to going towards the discussion of what are
the better ways of doing that, I think it is a
good idea to ask the question of: 

  When emacs24 is released and we say "emacs24 has bidi",
  
What does that mean?

Would support for R2L scripts in emacs24 be same as
L2R, in every respect and every where?

Or would emacs24 just support R2L in its core as
an editor? -- no goal of having any packages being
bidi-aware.

Of course, there is plenty of room between those
two extremes. In which case, there is the question
of: which packages?

As an R2L scripts user, my hope is for emacs24 at
a minimum to have:

  A- All emacs24 foundational packages be bidi-aware:
     dired, buffers lists, ...

  B- One usable set of basic personal communication/organization tools:
     Gnus, bbdb, calendar, org

As far as I can tell, making an emacs package
bidi-aware involves 2 things:

  1) Explicit over writing of paragraph direction
     where needed

  2) Custom formating because of bidi

For (2), here is an example of bidi in subject
line messing up the summary lines of Gnus.

---- Gnus Summary Sample Begin ----
:!   2 moons in the sky , August the 27th, يک آسمان و دو ماه|  37 |First LAST           |Jul 13 '10 
:!   Fw: بسیار جالب و خواندنی است ... فقط با دقت و حوصله بخو| 284 |First LAST           |14:15 
:!   Re: Next Saturday instead of this weekend              |  36 |First LAST           |14:22 
---- Gnus Summary Sample Ends  ----

The line number column for first message
[English+Farsi] (37) is out of place.

The line number column for second message
[Farsi] (284) is out of place.

For (1) paragraph direction detection, there may
be things that can be done in the core to improve
auto direction detection.

Making all of this happen is likely not going to
be easy because most (if not all) emacs package
developers are non-bidi-readers who likely don't
need or care about bidi.

So, I think we need to look for more serious bidi
users and pair them up with well selected package
owners.

For example, if Lars wanted to make the Gnus
Summary bidi-aware, I am there.

Your thoughts?

...Mohsen

...محسن

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Mohsen BANAN <list-general@mohsen.1.banan.byname.net>
> Cc: David Kastrup <dak@gnu.org>,  emacs-devel@gnu.org
> Date: Sun, 31 Jul 2011 18:14:41 -0700
> 
> ---- Gnus Summary Sample Begin ----
> :!   2 moons in the sky , August the 27th, يک آسمان و دو ماه|  37 |First LAST           |Jul 13 '10 
> :!   Fw: بسیار جالب و خواندنی است ... فقط با دقت و حوصله بخو| 284 |First LAST           |14:15 
> :!   Re: Next Saturday instead of this weekend              |  36 |First LAST           |14:22 
> ---- Gnus Summary Sample Ends  ----
> 
> The line number column for first message
> [English+Farsi] (37) is out of place.
> 
> The line number column for second message
> [Farsi] (284) is out of place.

Gnus should add an LRM character after the subject to get this right.

> For (1) paragraph direction detection, there may
> be things that can be done in the core to improve
> auto direction detection.

The detection algorithm follows the UAX#9.  What improvements do you
have in mind, and what are the use cases where these improvements will
do a better job than what we have now?

Re: bidi-display-reordering is now non-nil by default

[Mohsen BANAN]

>>>>> On Mon, 01 Aug 2011 05:54:24 +0300, Eli Zaretskii <eliz@gnu.org> said:

  Mohsen> For (1) paragraph direction detection, there may
  Mohsen> be things that can be done in the core to improve
  Mohsen> auto direction detection.

  Eli> The detection algorithm follows the UAX#9.  What improvements do you
  Eli> have in mind, and what are the use cases where these improvements will
  Eli> do a better job than what we have now?

The particular case I have in mind is where a
number of paragraphs get prefixed and then are
treated as one paragraph.

I am suggesting that the detection algorithm looks
for the likelyhood of a prefix being in the works
and then ignores the prefix and does detection as
if there were no prefixes.

The obvious example of such scenario is citations
in MUAs. However this is common enough that I
think considerations of doing it in the core is
reasonable.

Here is an example:

---

This is paragraph 1 all in english.

اين پاراگراف ۲ تماما به فارسى است.

This is paragraph 3 which is mixed with فارسى and english.

اين پاراگراف ۴ مخلوت English و فارسى است.



  >>>
  >>> This is paragraph 1 all in english.
  >>>
  >>> اين پاراگراف ۲ تماما به فارسى است.
  >>>
  >>> This is paragraph 3 which is mixed with فارسى and english.
  >>>
  >>> اين پاراگراف ۴ مخلوت English و فارسى است.
  >>>

---

Note all 4 paragraphs' directionality are detected
fine based on UAX#9 prior to citation.

Once cited/prefixed they all become 1 paragraph.

The communication ramifications on paragraph 2 are
minor. The reader would get it. 

But communication ramifications on paragraph 4 are
very problematic. The sentence becomes
incomprehensible/mangled.

So, here is what I am suggesting:

 - We add an additional value of 'fancy to
   bidi-paragraph-direction making it:
    nil, left-to-right, right-to-left, fancy

 - If bidi-paragraph-direction is 'fancy,
    
     - do prefix guessing
     - ignore prefix
     - do UAX#9
     - re-insert prefix

This will likely help with simple tabularization
in addition to mail citations.

What do you think?

...Mohsen

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Mohsen BANAN <list-general@mohsen.1.banan.byname.net>
> Cc: Mohsen BANAN <list-general@mohsen.1.banan.byname.net>,  emacs-devel@gnu.org
> Date: Tue, 02 Aug 2011 19:39:08 -0700
> 
>   >>>
>   >>> This is paragraph 1 all in english.
>   >>>
>   >>> اين پاراگراف ۲ تماما به فارسى است.
>   >>>
>   >>> This is paragraph 3 which is mixed with فارسى and english.
>   >>>
>   >>> اين پاراگراف ۴ مخلوت English و فارسى است.
>   >>>
> 
> ---
> 
> Note all 4 paragraphs' directionality are detected
> fine based on UAX#9 prior to citation.
> 
> Once cited/prefixed they all become 1 paragraph.
> 
> The communication ramifications on paragraph 2 are
> minor. The reader would get it. 
> 
> But communication ramifications on paragraph 4 are
> very problematic. The sentence becomes
> incomprehensible/mangled.

Yes, I agree.

> So, here is what I am suggesting:
> 
>  - We add an additional value of 'fancy to
>    bidi-paragraph-direction making it:
>     nil, left-to-right, right-to-left, fancy
> 
>  - If bidi-paragraph-direction is 'fancy,
>     
>      - do prefix guessing
>      - ignore prefix
>      - do UAX#9
>      - re-insert prefix
> 
> This will likely help with simple tabularization
> in addition to mail citations.
> 
> What do you think?

This is too complicated.  Paragraph detection is at the lowest level
of the reordering engine, so it must be simple and fast.  We saw just
yesterday a case of severe slowdown of redisplay because paragraph
detection was working too hard.

I have an idea that would be much easier to implement: ignore neutral
and weak characters for the purposes of the decision where a paragraph
ends.  This can be done by modifying the regexp used to detect the
paragraph separator.  This would make all the citations preserve their
original paragraphs, notwithstanding the citation.  Then it will look
like this (but without the added empty lines, of course):

  >>>
  >>> This is paragraph 1 all in english.
  >>>

  >>> اين پاراگراف ۲ تماما به فارسى است.
  >>>

  >>> This is paragraph 3 which is mixed with فارسى and english.
  >>>

  >>> اين پاراگراف ۴ مخلوت English و فارسى است.
  >>>

(I assume you are looking at this in Emacs 24, which will reorder
correctly.)  Is this okay?

Anyway, this feature will probably have to wait until Emacs 24.2,
because there are more serious issues on the table, and we want to
start the pretest for Emacs 24.1 ASAP.  Modifications of paragraph
separator is something that needs to be thoroughly tested before it
hits the end users' machines, because it can potentially destabilize
the display engine to a great extent.

For now, users will have to learn to insert empty lines ;-)

Re: bidi-display-reordering is now non-nil by default

[Lars Ingebrigtsen]

Mohsen BANAN <list-general@mohsen.1.banan.byname.net> writes:

> For example, if Lars wanted to make the Gnus
> Summary bidi-aware, I am there.

I definitely want that, and if you can help make this work better, that
would be great.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: bidi-display-reordering is now non-nil by default

[Mohsen BANAN]

>>>>> On Mon, 01 Aug 2011 17:51:55 +0200, Lars Ingebrigtsen <larsi@gnus.org> said:

  Lars> Mohsen BANAN <list-general@mohsen.1.banan.byname.net> writes:
  >> For example, if Lars wanted to make the Gnus
  >> Summary bidi-aware, I am there.

  Lars> I definitely want that, and if you can help make this work better, that
  Lars> would be great.

Great!

For making Gnus bidi-aware consider us paired.

Of course I can be a tester and identify needed
features.

But beyond that, with your help I may be able to
do the enhancements -- if you prefer that route.
The benefit of involving me in the enhancements is
that I can then spread bidi-awareness into other
packages.

The low hanging fruit right now is:

  1) Adding an LRM character after the subject
     line in the Suummary buffer.

  2) Adding an LRM character after the From name
     in the Summary buffer.

  3) Fix paragraph detection in cited emails
     (Article buffer). Please see my suggestion to
     Eli about doing that in the core.

Thanks.

...Mohsen

Re: bidi-display-reordering is now non-nil by default

[Lars Magne Ingebrigtsen]

Mohsen BANAN <list-general@mohsen.1.banan.byname.net> writes:

> The low hanging fruit right now is:
>
>   1) Adding an LRM character after the subject
>      line in the Suummary buffer.
>
>   2) Adding an LRM character after the From name
>      in the Summary buffer.

As I said earlier, I think adding characters to the text to control pure
layout issues is a sub-optimal design decision.  It's like adding VT
control characters to the text to colourise it.  Why not control this
stuff with text properties, like we do with (almost) anything else
that's similar to this?

LRM characters seem like a neat hack for things like the HELLO file,
where you have a plain text file.  But for stuff like the summary
buffer, you're basically just concerned with lining stuff up properly.
Adding special characters here doesn't seem warranted to me.

And for those who forgot the two options we have with LRM:

1) make the LRM invisible and intangible, which means that if somebody
cuts and pastes a (seemingly) ASCII summary buffer, it'll be
(inexplicably) utf-8 with LRM characters interspersed after saving or
mailing it, or

2) have the LRM characters visible, which means that pushing <Right>
over a line will have an inexplicable "stop" several times over the
line.

Please correct me if this isn't what all y'all have in mind.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Date: Wed, 03 Aug 2011 20:45:16 +0200
> Cc: emacs-devel@gnu.org
> 
> Mohsen BANAN <list-general@mohsen.1.banan.byname.net> writes:
> 
> > The low hanging fruit right now is:
> >
> >   1) Adding an LRM character after the subject
> >      line in the Suummary buffer.
> >
> >   2) Adding an LRM character after the From name
> >      in the Summary buffer.
> 
> As I said earlier, I think adding characters to the text to control pure
> layout issues is a sub-optimal design decision.

You are wrong.  LRM, RLM, etc. exist _precisely_ for these purposes:
to arrange bidirectional text for display as the display designer
wishes, where the implicit reordering mandated by the UBA does not
give satisfactory results.  These _explicit_ directional control
characters exist for when the _implicit_ reordering fails.  It's the
functional equivalent of the corresponding HTML directives -- you
won't say that HTML is "suboptimal" when it dictates to the browser
how to render bidirectional text, would you?

> Why not control this stuff with text properties, like we do with
> (almost) anything else that's similar to this?

Because (a) text properties are specific to Emacs, and (b) they cannot
overlap (for the same property).  By contrast, to force certain visual
order, one must sometimes force some direction on a portion of text
and then the opposite direction on an inner substring of that very
text.  Text properties won't grok that.

This issue was discussed at length long ago, when the basic design of
bidi support for Emacs was on the table.  Text properties were
suggested almost immediately, then rejected after prolonged debates,
because they simply aren't up to this job, and because the Unicode
Standard already tells us how to DTRT, we just need to heed.

> LRM characters seem like a neat hack for things like the HELLO file,
> where you have a plain text file.  But for stuff like the summary
> buffer, you're basically just concerned with lining stuff up properly.
> Adding special characters here doesn't seem warranted to me.

What is the difference between aligning HELLO and aligning a summary
buffer?  They are both plain text, and they both are arranged to align
nicely.

> And for those who forgot the two options we have with LRM:
> 
> 1) make the LRM invisible and intangible, which means that if somebody
> cuts and pastes a (seemingly) ASCII summary buffer, it'll be
> (inexplicably) utf-8 with LRM characters interspersed after saving or
> mailing it, or
> 
> 2) have the LRM characters visible, which means that pushing <Right>
> over a line will have an inexplicable "stop" several times over the
> line.

The latter is not true: a visible LRM displays as a thin space, so the
cursor is not stuck.

> Please correct me if this isn't what all y'all have in mind.

Using LRM/RLM _is_ TRT, Lars.  It may sound weird to you, but it's
like democracy: the worst political system, except all the rest.  I
wrote the entire Hebrew tutorial with these characters, and didn't see
any problems.  It just takes some time to get used to it, that's all.
R2L users already are used to it.

Re: bidi-display-reordering is now non-nil by default

[Stephen J. Turnbull]

Eli Zaretskii writes:
 > > From: Lars Magne Ingebrigtsen <larsi@gnus.org>

 > > As I said earlier, I think adding characters to the text to control pure
 > > layout issues is a sub-optimal design decision.
 > 
 > You are wrong.  LRM, RLM, etc. exist _precisely_ for these purposes:
 > to arrange bidirectional text for display as the display designer
 > wishes, where the implicit reordering mandated by the UBA does not
 > give satisfactory results.  These _explicit_ directional control
 > characters exist for when the _implicit_ reordering fails.  It's the
 > functional equivalent of the corresponding HTML directives -- you
 > won't say that HTML is "suboptimal" when it dictates to the browser
 > how to render bidirectional text, would you?

Eli, you seem to forget that *Unicode is a wire protocol*, an inter-
application communication tool.  It is not intended to be a
specification, or even recommendation, of how applications handle text
internally.

More specifically, everything you write applies to *streams of Unicode
(or HTML) text communicated between applications* (eg, files or HTTP
streams), as Lars points out.  I don't know enough about the details
to have an opinion on what *should* be done internally by Emacs, but
your argument above is bogus.  Lars's example of ANSI color control
shows why.  Lars's arguments about the deficiencies of including these
*control* characters in the text are precisely the same as the
arguments for deprecating the "Plane 14 language tags."  (Yes, I
understand why directional marks will *not* be deprecated in that
way by the Unicode standard.  Nevertheless, the arguments are valid
and should be given some weight for *internal* implementation.)

Of course on writing a stream to the outside world, Emacs will need to
use directional marks.  Surely Lars does not deny that!  However,
internally, text properties could in theory suffice, just as they do
for ANSI coloring.

 > Because (a) text properties are specific to Emacs, and (b) they cannot
 > overlap (for the same property).  By contrast, to force certain visual
 > order, one must sometimes force some direction on a portion of text
 > and then the opposite direction on an inner substring of that very
 > text.  Text properties won't grok that.

Huh?  Of course text properties nest.  If for some subtle reason, they
don't quite nest correctly for this purpose, overlays most likely will.

I really don't understand your arguments here.

 > What is the difference between aligning HELLO and aligning a summary
 > buffer?  They are both plain text, and they both are arranged to align
 > nicely.

HELLO arrives as an external plain text stream, and therefore is
governed by the Unicode standard.  The summary buffer is constructed
by Emacs and it is not plain text (it has a *lot* of structure, being
mousable etc), and therefore is not governed by the standard for its
*implementation*.  It *is* governed by the standard for the *display*,
because that is I/O.

IOW, you may be right, but please don't use incorrect arguments like
the ones you have used in this post.  It's just going to confuse
people, and bidi is hard enough already.

 > Using LRM/RLM _is_ TRT, Lars.  It may sound weird to you, but it's
 > like democracy: the worst political system, except all the rest. 

Well, you're the expert, which counts for a lot (and IIRC Mohsen seems
to agree with you, which adds even more credibility).  Nevertheless,
Lars (and the Gnus crews in general) are probably the leading experts
on Emacs edge cases.  You would do well to consult his (their)
expertise on issues like copy/paste and wacky behavior of special text
properties and characters, etc.

 > I wrote the entire Hebrew tutorial with these characters, and didn't see
 > any problems.

Sure, because you started with the intention of producing a plain text
stream conforming to Unicode while implementing correct direction of
exceptional intervals.  But ....  How much copy/pasting did you do?
How many directional marks are needed in the Hebrew TUTORIAL, given
the full BIDI algorithm implementation, and how many are redundant?
Have you copied portions of the TUTORIAL with embedded marks into
email headers and gotten appropriate results?

I bet that, as Lars implies, Emacs is going to need
`yank-dropping-directional-marks' in some applications.  Maybe even
`yank-dropping-directional-marks-at-interval-boundaries' or similar.

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Cc: Lars Magne Ingebrigtsen <larsi@gnus.org>,
>     list-general@mohsen.1.banan.byname.net,
>     emacs-devel@gnu.org
> Date: Thu, 04 Aug 2011 12:23:28 +0900
> 
> Eli, you seem to forget that *Unicode is a wire protocol*, an inter-
> application communication tool.  It is not intended to be a
> specification, or even recommendation, of how applications handle text
> internally.

Yes, but we are not talking about the internal handling.  We are
talking about display, which is an external, user-visible part of the
issue.  The Unicode Bidirectional Algorithm is a specification for
converting a stream of text into an array of character glyphs on the
screen.  It is not a wire protocol.  Nor is it specific to text
external to Emacs: after all, the internal storage of text in Emacs,
as in many other applications, is just a linear byte stream.

Let's go back to the issue at hand: the directional control
characters.  A quote from UAX#9:

  [...] there are circumstances where an implicit bidirectional
  ordering is not sufficient to produce comprehensible text. To deal
  with these cases, a minimal set of directional formatting codes is
  defined to control the ordering of characters when rendered. This
  allows exact control of the display ordering for legible interchange
  and ensures that plain text used for simple items like filenames or
  labels can always be correctly ordered for display.

  The directional formatting codes are used only to influence the
  display ordering of text. [...]

> Of course on writing a stream to the outside world, Emacs will need to
> use directional marks.  Surely Lars does not deny that!  However,
> internally, text properties could in theory suffice, just as they do
> for ANSI coloring.

This option (converting directional marks in external stream to some
Emacs feature on I/O) was also discussed at the time (nearly 10 years
ago).  It is possible to implement it, but it is unnecessarily
complicated, and it even has some hard-to-resolve issues.  For
example, what if the user inserts these characters manually? we will
then face a very real risk to introduce subtle bugs whereby saving the
text to a disk file, then visiting that file could produce a buffer
whose contents are different.  Such unexpected conversions behind
user's back proved to be an annoyance, as the experience of MULE
shows.

>  > Because (a) text properties are specific to Emacs, and (b) they cannot
>  > overlap (for the same property).  By contrast, to force certain visual
>  > order, one must sometimes force some direction on a portion of text
>  > and then the opposite direction on an inner substring of that very
>  > text.  Text properties won't grok that.
> 
> Huh?  Of course text properties nest.

A single character can have only one property of each type.  Let's say
we call this property `direction' and give it 2 values: L2R and R2L.
Then imagine the following scenario:

  . you mark a portion of text with L2R direction property

  . you then want to mark part of that portion with R2L direction
    property (there are situations where this is necessary, I can show
    examples if this matters, but for now please believe me)

  . since each character can have only one value of the direction
    property, you cannot do this in any simple way; you'd need to
    split the original region in 3 parts, which is ugly and
    complicates what needs to be done when text in this region is
    deleted (keep in mind that the UBA mandates support of up to 60
    levels of such embedded direction reversals, don't ask me why, and
    Emacs is in full compliance)

> If for some subtle reason, they don't quite nest correctly for this
> purpose, overlays most likely will.

Overlays don't get copied with the text, so if you copy/paste text
into another area of the same buffer or into another buffer, the nice
display will be lost.  We could complicate the heck out of yanking so
it reinserts the overlays, of course, but why complicate things if an
easier way is available that is straightforward?

>  > What is the difference between aligning HELLO and aligning a summary
>  > buffer?  They are both plain text, and they both are arranged to align
>  > nicely.
> 
> HELLO arrives as an external plain text stream, and therefore is
> governed by the Unicode standard.  The summary buffer is constructed
> by Emacs and it is not plain text

But it should be possible to copy portions of that buffer elsewhere,
and such a copy should keep its visual appearance on the screen with
minimal fuss, or else users will be annoyed.  Right?  The question
that bugged us during the early stages of the design was how do you
ensure this without asking Lisp application programmers to jump
through the hoops every time text is copied or saved or read.  It
turns out that using the directional control characters is the easiest
way.

> (it has a *lot* of structure, being mousable etc), and therefore is
> not governed by the standard for its *implementation*.

It's not governed by the standard, but following the standard is the
easiest way of achieving the goal with minimal implications.

> How many directional marks are needed in the Hebrew TUTORIAL, given
> the full BIDI algorithm implementation

Not many, but some.  About 120, if my count is correct.

> and how many are redundant?

None.  I used them only where the normal implicit reordering didn't
yield the correct display.

> Have you copied portions of the TUTORIAL with embedded marks into
> email headers and gotten appropriate results?

Yes.  It works, and works seamlessly.  That's the whole point of using
these control characters.

> I bet that, as Lars implies, Emacs is going to need
> `yank-dropping-directional-marks' in some applications.

If we drop the marks on yanking, text will look differently when
yanked, sometimes completely differently, to the degree of being
incomprehensible.  I think that way lies madness, if we want a decent
support of bidi scripts.  So such a feature would be ill-advised, and
I will do my best to convince people out of it.

Re: bidi-display-reordering is now non-nil by default

[Kenichi Handa]

In article <E1QoqI3-0004zu-Ij@fencepost.gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> Yes, but we are not talking about the internal handling.  We are
> talking about display, which is an external, user-visible part of the
> issue.  The Unicode Bidirectional Algorithm is a specification for
> converting a stream of text into an array of character glyphs on the
> screen.  It is not a wire protocol.  Nor is it specific to text
> external to Emacs: after all, the internal storage of text in Emacs,
> as in many other applications, is just a linear byte stream.

> Let's go back to the issue at hand: the directional control
> characters.  A quote from UAX#9:

>   [...] there are circumstances where an implicit bidirectional
>   ordering is not sufficient to produce comprehensible text. To deal
>   with these cases, a minimal set of directional formatting codes is
>   defined to control the ordering of characters when rendered. This
>   allows exact control of the display ordering for legible interchange
>   and ensures that plain text used for simple items like filenames or
>   labels can always be correctly ordered for display.

>   The directional formatting codes are used only to influence the
>   display ordering of text. [...]

UAX#9's algorithm is for a paragraph, and as for paragraph,
it says:

  The algorithm reorders text only within a paragraph;
  characters in one paragraph have no effect on characters
  in a different paragraph. Paragraphs are divided by the
  Paragraph Separator or appropriate Newline Function (for
  guidelines on the handling of CR, LF, and CRLF, see
  Section 4.4, Directionality, and Section 5.8, Newline
  Guidelines of [Unicode]). Paragraphs may also be
  determined by higher-level protocols: for example, the
  text in two different cells of a table will be in
  different paragraphs.

The case of summary line belongs to the last sentence above.
In a summary line, we surely have multiple cells, and
UAX#9's algorithm should be applied to each cell
independently.

So what we need is a way to tell the display engine about
cells themselves or about cell-separators (or artificial
paragraph separators).

To display something like summarly line, we always use
separating character(s) (tab, space, |, ...).  How about
putting the text property `paragraph-separator' to them, and
making handle_stop (in xdisp.c) to pay attention to that
property too.

---
Kenichi Handa
handa@m17n.org

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Kenichi Handa <handa@m17n.org>
> Cc: stephen@xemacs.org, larsi@gnus.org, list-general@mohsen.1.banan.byname.net,
>         emacs-devel@gnu.org
> Date: Thu, 04 Aug 2011 15:55:20 +0900
> 
>   The algorithm reorders text only within a paragraph;
>   characters in one paragraph have no effect on characters
>   in a different paragraph. Paragraphs are divided by the
>   Paragraph Separator or appropriate Newline Function (for
>   guidelines on the handling of CR, LF, and CRLF, see
>   Section 4.4, Directionality, and Section 5.8, Newline
>   Guidelines of [Unicode]). Paragraphs may also be
>   determined by higher-level protocols: for example, the
>   text in two different cells of a table will be in
>   different paragraphs.
> 
> The case of summary line belongs to the last sentence above.

That is one possible interpretation, yes.  But it isn't the only one.
The goal is to have correct display of the summary lines, not to
decide whether we are dealing with cells or not.  Directional control
character allow to render summary lines for display correctly even
without "cells".

> In a summary line, we surely have multiple cells, and
> UAX#9's algorithm should be applied to each cell
> independently.
> 
> So what we need is a way to tell the display engine about
> cells themselves or about cell-separators

It's possible, but it will need changes of a different kind in Gnus
(and in other modes that produce similar display).  See below.

> (or artificial paragraph separators).

This alternative, I don't recommend, for the reasons explained below.

> To display something like summarly line, we always use
> separating character(s) (tab, space, |, ...).  How about
> putting the text property `paragraph-separator' to them, and
> making handle_stop (in xdisp.c) to pay attention to that
> property too.

Such a text property is not needed: the UBA already provides a
character whose function is "segment separator": the TAB character.
Here's a demonstration of it in action in Emacs, a single line with 5
fields or "cells" (you need Emacs 24 with bidi-display-reordering
turned on to see it rendered correctly):

    abcde 	אבגדה 	12345 	עבודה 	9876

To see how this is rendered by default, without segment separators,
remove the TAB characters between the different fields, leaving just
the blanks:

    abcde אבגדה 12345 עבודה 9876

So we can have such "cells" already, but that requires to add TAB
characters to separate fields in the summary buffers, and I already
hear Lars telling me that this is "a hack a-la VT color escape
sequences".  After all, a TAB is another control character...

TABs also have a disadvantage that the tab stops can be customized,
and the visual appearance depends on the indentation of the first
character in the line.  Whether this is better than LRM, I don't know.

As for "artificial paragraph separators": I don't want to bother
people with too many technicalities (interested readers can read
relevant portions of bidi.c), but supporting "paragraphs" that have no
relation to newlines, i.e. begin and end not on line boundaries, would
mean a significant surgery of the current reordering engine.  So even
if this is deemed as the optimal solution (and I'm not at all
convinced it is), it won't make it in time for Emacs 24.1, at the very
least.

Actually, any solution based on special text properties has a similar
problem: bidi.c works below handle_stop, and ignores text properties
entirely (with the single exception of `display' properties that
replace the underlying text with something else, like images or
display strings).  So any such solution will also be out of the
question for Emacs 24.1.

So eventually, this discussion boils down to this: do we want Gnus and
other Emacs-based MUAs to have reasonable support in their summary
buffers for R2L scripts in Emacs 24.1, or don't we?  If we do (and I
surely hope so), then we must use only the facilities available now,
which include (1) LRM characters and (2) TABs to separate the "cells".
Anything else will simply not work in Emacs 24.1.

Re: bidi-display-reordering is now non-nil by default

[Kenichi Handa]

In article <E1QouuP-0006aC-L8@fencepost.gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> >   Guidelines of [Unicode]). Paragraphs may also be
> >   determined by higher-level protocols: for example, the
> >   text in two different cells of a table will be in
> >   different paragraphs.
> > 
> > The case of summary line belongs to the last sentence above.

> That is one possible interpretation, yes.  But it isn't the only one.

I think it is the most natural/appropriate interpretation.

> The goal is to have correct display of the summary lines, not to
> decide whether we are dealing with cells or not.  Directional control
> character allow to render summary lines for display correctly even
> without "cells".

They can, but it is better to avoid using them to workaround
the "cell" problem.

> As for "artificial paragraph separators": I don't want to bother
> people with too many technicalities (interested readers can read
> relevant portions of bidi.c), but supporting "paragraphs" that have no
> relation to newlines, i.e. begin and end not on line boundaries, would
> mean a significant surgery of the current reordering engine.  So even
> if this is deemed as the optimal solution (and I'm not at all
> convinced it is), it won't make it in time for Emacs 24.1, at the very
> least.

> Actually, any solution based on special text properties has a similar
> problem: bidi.c works below handle_stop, and ignores text properties
> entirely (with the single exception of `display' properties that
> replace the underlying text with something else, like images or
> display strings).  So any such solution will also be out of the
> question for Emacs 24.1.

I think we should not stop finding the right solution just
to release 24.1 on time.  Isn't it worth delaying 24.1 for
the right solution?

---
Kenichi Handa
handa@m17n.org

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Kenichi Handa <handa@m17n.org>
> Cc: stephen@xemacs.org, larsi@gnus.org, list-general@mohsen.1.banan.byname.net,
>         emacs-devel@gnu.org
> Date: Tue, 09 Aug 2011 15:11:41 +0900
> 
> In article <E1QouuP-0006aC-L8@fencepost.gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:
> 
> > >   Guidelines of [Unicode]). Paragraphs may also be
> > >   determined by higher-level protocols: for example, the
> > >   text in two different cells of a table will be in
> > >   different paragraphs.
> > > 
> > > The case of summary line belongs to the last sentence above.
> 
> > That is one possible interpretation, yes.  But it isn't the only one.
> 
> I think it is the most natural/appropriate interpretation.

I'm not convinced.  However, as I wrote, if we are okay with using
TABs to separate fields in summary buffers, the necessary
infrastructure is already implemented.  Personally, I'm not sure that
using TABs is the best way, for reasons I described in the quoted
message, but I'll leave this decision to Gnus developers and users.
Maybe making the TABs invisible (with the `invisible' text property)
will help us maintain a plausible display with predictable format, and
thus resolve the difficulties that bother me.

> > The goal is to have correct display of the summary lines, not to
> > decide whether we are dealing with cells or not.  Directional control
> > character allow to render summary lines for display correctly even
> > without "cells".
> 
> They can, but it is better to avoid using them to workaround
> the "cell" problem.

Why?  I didn't yet hear a single explanation of why people don't like
them.  They are just formatting control characters, like TAB, SPC, and
NL.

Without providing some good reasons for staying away of the
directional control characters, all this discussion is just waste of
energy and bandwidth.  Unless someone else is going to implement the
proposed text property alternative, I will need a _very_ good reason
to throw away what cost me many months of hard work and do it again
differently, because as far as I'm concerned, what we have now is very
satisfactory, to say the least, and no serious attempts were made this
far to use them to solve this issue.

> I think we should not stop finding the right solution just
> to release 24.1 on time.  Isn't it worth delaying 24.1 for
> the right solution?

How long are you prepared to wait?  How long are Emacs users prepared
to wait?

First, "the right solution" is not well defined yet.  People are
proposing to use text properties, but the proposal itself is not
detailed enough.  For example, are we going to use text properties for
any other bidi-related features, and if so, which ones?  We had better
have a good and detailed answer to this and similar questions, because
making such deep changes (see below) without a clear plan, or, worse,
just for the benefit of a single use case (lines with fields) is a bad
idea: it will complicate the heck out of the display code, which is
already a hard to maintain monstrosity, where alternatives already
exist.

I see no hope of coming up with a reasonably complete proposal from
this discussion, because most of the people arguing for the change
don't use R2L scripts and have no clear idea what is needed and how it
will affect users.  So it is again up to me, and I don't see the need
for such a change to begin with, because no one cared to tell what is
so dead wrong with the current facilities, which, btw, were not chosen
at random, but came out of long discussions and careful study of the
needs and of "prior art".

Next, implementing this suggestion is not a simple task at all.
Forcing bidi.c to check text properties will at best slow down the
display engine, which is already slower than Emacs 23.  Worse, there
are many potential pitfalls here, because code that examines text
properties will need to avoid walking the buffer, or it will cause
infinite recursion.  In the current implementation, at least handling
of the `display' and `invisible' properties does involve walking the
buffer--these parts will have to be redesigned and reimplemented.
Handlers of other properties will have to be carefully examined, to
make sure they don't run the same risk; in the current design, I never
needed such an analysis because of complete separation between
scanning the buffer and applying text properties.

Last, but not least, the elaborate logic that finds and applies text
properties when the next character in the visual order can be far away
of the current one will have to be redesigned and reimplemented.
Getting that logic right took months of trial and crash, because the
intricate details of how all this works in the display engine are not
documented anywhere.  I still bump into surprises from time to time to
this day, when I step through set_iterator_to_next and its
subroutines.  Who will have the energy and motivation to go through
all that process again?

Re: bidi-display-reordering is now non-nil by default

[Stephen J. Turnbull]

Eli Zaretskii writes:

 > Yes, but we are not talking about the internal handling.

Of course we are.  Whether explicit directionality is represented in
the buffer as directional marks or as a text property is a purely
internal matter.

 > Nor is it specific to text external to Emacs: after all, the
 > internal storage of text in Emacs, as in many other applications,
 > is just a linear byte stream.

That's just saying that you *can* use directional marks for this
purpose.  It is not a reason you *should*.

 > Such unexpected conversions behind user's back proved to be an
 > annoyance, as the experience of MULE shows.

With all due respect to Handa-san and the other MULE implementers,
MULE was implemented to be too smart for its own good.  This is a good
idea for Japanese text, where at this very second I can lay my hands
on files containing no less than 5 major very different encodings of
Japanese (not to mention at least 5 more minor corporate variations,
plus Japanese written in scripts meant for Chinese).  It was not such
a great idea for other environments, although it mostly doesn't hurt
too much.

I would not extrapolate from the MULE experience to designs that could
be a lot more precise and explicit.

 >   . since each character can have only one value of the direction
 >     property, you cannot do this in any simple way; you'd need to
 >     split the original region in 3 parts, which is ugly

It's no uglier than inserting control sequences, which means that text
has yet another feature that prevents it from being an array of
characters.

 >     and complicates what needs to be done when text in this region
 >     is deleted (keep in mind that the UBA mandates support of up to
 >     60 levels of such embedded direction reversals, don't ask me
 >     why, and Emacs is in full compliance)
 > 
 > > If for some subtle reason, they don't quite nest correctly for this
 > > purpose, overlays most likely will.
 > 
 > Overlays don't get copied with the text, so if you copy/paste text
 > into another area of the same buffer or into another buffer, the nice
 > display will be lost.

Ah.  I see.  I forgot about that "feature" of the implementation in
Emacs.

 > We could complicate the heck out of yanking so it reinserts the
 > overlays, of course, but why complicate things if an easier way is
 > available that is straightforward?

Because it's correct, mostly.  You have convinced me that it would be
quite inconvenient to implement these things in Emacs without
introducing XEmacs-like extents that can have some of the properties
of both text properties and overlays.  I'm still not convinced that
the problems Lars worries about won't annoy the users.

 > The question that bugged us during the early stages of the design
 > was how do you ensure this without asking Lisp application
 > programmers to jump through the hoops every time text is copied or
 > saved or read.  It turns out that using the directional control
 > characters is the easiest way.

That's reasonable.

 > If we drop the marks on yanking, text will look differently when
 > yanked, sometimes completely differently, to the degree of being
 > incomprehensible.

Obviously.  Nevertheless, I suspect there will be applications where
that is desirable (especially for marks at the boundary of the yanked
region).

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Cc: larsi@gnus.org,
>     list-general@mohsen.1.banan.byname.net,
>     emacs-devel@gnu.org
> Date: Thu, 04 Aug 2011 19:04:22 +0900
> 
>  > Nor is it specific to text external to Emacs: after all, the
>  > internal storage of text in Emacs, as in many other applications,
>  > is just a linear byte stream.
> 
> That's just saying that you *can* use directional marks for this
> purpose.  It is not a reason you *should*.

Directional marks need to be supported anyway, if we want to be
compliant.  Using that for "fixing" specially-formatted text as well
gives us 2 features for the cost of one.

> I would not extrapolate from the MULE experience to designs that could
> be a lot more precise and explicit.

Precisely and explicitly, I think that using lossy conversions is not
a good idea, if there are practical alternatives.  Just visiting a
file, then saving it without changes should produce the a byte stream
that is identical to the original one.  I hope you agree with that.

>  >   . since each character can have only one value of the direction
>  >     property, you cannot do this in any simple way; you'd need to
>  >     split the original region in 3 parts, which is ugly
> 
> It's no uglier than inserting control sequences

I mean ugly on the implementation level.  This splitting (and the
corresponding joining) need to be implemented, or else each Lisp
program would need to do it by hand.  By contrast, inserting a
character does not need any additional features.  Text in mail summary
buffers is already formatted specially, by inserting separators
between fields; adding one more character to the field separator is
not fundamentally different.

> which means that text has yet another feature that prevents it from
> being an array of characters.

??? LRM is just another character, like SPC or `|'.  The text is still
an array of characters after inserting it.

> I'm still not convinced that the problems Lars worries about won't
> annoy the users.

I'm not convinced either.  But "the proof of the pudding is in
eating."  These questions are around since 12 years ago, and no one
ever had the definitive answers.  If I'd to wait for such answers,
Emacs wouldn't have bidi support it has now.  So I made some decisions
(and published them here almost 2 years ago, btw), and implemented
them.  Having used them for some time, I can say that they are quite
satisfactory, but that's me.

Let's get Emacs 24.1 out the door, using the existing functionality,
and let the users of R2L scripts at large judge.  This discussion,
however useful and enlightening, cannot be a substitute for user
experience, especially since most of the participants don't use R2L
scripts.

>  > If we drop the marks on yanking, text will look differently when
>  > yanked, sometimes completely differently, to the degree of being
>  > incomprehensible.
> 
> Obviously.  Nevertheless, I suspect there will be applications where
> that is desirable (especially for marks at the boundary of the yanked
> region).

I don't see why, but it isn't worth arguing, since removing them is
easy.

Re: bidi-display-reordering is now non-nil by default

[Stephen J. Turnbull]

Eli Zaretskii writes:

 > Precisely and explicitly, I think that using lossy conversions is not
 > a good idea, if there are practical alternatives.  Just visiting a
 > file, then saving it without changes should produce the a byte stream
 > that is identical to the original one.  I hope you agree with that.

I don't.  Not in the context of Unicode.  It should produce a sequence
of characters identical to the original one.  However, the user may
have excellent reasons for doing, say, normalization, and therefore
the sequence of bytes may be different.  Applications should be robust
to that, and I'm not terribly sympathetic to ones that aren't.

I do grant that it will often be what the user wants, especially Emacs
users, since Emacs can provide a byte-level view of the file if the
user wants that.  On the other hand, I see no reason why a
text-property-based implementation should be lossy.  Handling that
kind of thing is done all the time in serialization functions.

Regarding handling the splitting and joining of regions:

 > I mean ugly on the implementation level.  This splitting (and the
 > corresponding joining) need to be implemented,

I don't understand.  If `put-text-property' and friends don't get that
right already, more than bidi is in trouble, I should think.  What's
special about bidi?

 > So I made some decisions (and published them here almost 2 years
 > ago, btw), and implemented them.  Having used them for some time, I
 > can say that they are quite satisfactory, but that's me.

Yes, that is you.  But you're right, you have something that works for
you, more, you have no reason to believe that there's any crippling
bug in the design so it "should" work for most people who need it, and
it's time to get it out the door.

Re: bidi-display-reordering is now non-nil by default

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> Eli Zaretskii writes:
>
>  > Precisely and explicitly, I think that using lossy conversions is not
>  > a good idea, if there are practical alternatives.  Just visiting a
>  > file, then saving it without changes should produce the a byte stream
>  > that is identical to the original one.  I hope you agree with that.
>
> I don't.  Not in the context of Unicode.  It should produce a sequence
> of characters identical to the original one.

Not having the same sequence of bytes is painful.  There is a reason we
aimed for that with MULE when possible (some Japanese coding systems
don't make it possible) and with utf-8.

However, that is utterly orthogonal to the question how to work with
specially displayed/generated material like mode lines, group summaries,
help texts and so on.

For example, Emacs has a somewhat quaint mechanism called "fields".  It
would not make much sense to let directionality bleed over field borders
into neighbouring neutral characters, effectively moving them visually
across the field.

-- 
David Kastrup

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Thu, 04 Aug 2011 16:04:45 +0200
> 
> For example, Emacs has a somewhat quaint mechanism called "fields".  It
> would not make much sense to let directionality bleed over field borders
> into neighbouring neutral characters, effectively moving them visually
> across the field.

I'm not sure I understand.  Can you show an example where
directionality could "bleed" like that, or tell me how to create such
an example?  I'd like to see if there's some potential problem here.

TIA

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Cc: larsi@gnus.org,
>     list-general@mohsen.1.banan.byname.net,
>     emacs-devel@gnu.org
> Date: Thu, 04 Aug 2011 22:55:57 +0900
> 
> Eli Zaretskii writes:
> 
>  > Precisely and explicitly, I think that using lossy conversions is not
>  > a good idea, if there are practical alternatives.  Just visiting a
>  > file, then saving it without changes should produce the a byte stream
>  > that is identical to the original one.  I hope you agree with that.
> 
> I don't.  Not in the context of Unicode.  It should produce a sequence
> of characters identical to the original one.  However, the user may
> have excellent reasons for doing, say, normalization, and therefore
> the sequence of bytes may be different.

Normalization is not relevant to directional control characters.
Let's stick to the issue at hand: do you consider it a good idea to
remove or add these characters in an otherwise unmodified buffer?

> I see no reason why a text-property-based implementation should be
> lossy.

Because the user could type directional controls, and there's no way
for Emacs to know at all levels which one is to be treated in which
way.

> Regarding handling the splitting and joining of regions:
> 
>  > I mean ugly on the implementation level.  This splitting (and the
>  > corresponding joining) need to be implemented,
> 
> I don't understand.  If `put-text-property' and friends don't get that
> right already, more than bidi is in trouble, I should think.  What's
> special about bidi?

What is special is the fact that bidi needs nested regions with
different values for the same property.  Normally, if you put a
property with a value on a portion of text that has another value for
that property, the new value replaces the old one.

Re: bidi-display-reordering is now non-nil by default

[Stephen J. Turnbull]

Eli Zaretskii writes:

 > Let's stick to the issue at hand: do you consider it a good idea to
 > remove or add these characters in an otherwise unmodified buffer?

Er, you're the one who keep bringing up random irrelevancies like
byte-level equality.

And your question makes no sense.  In your implementation, they will
be present as characters in the buffer, and they should neither be
removed nor added per the Unicode standard.  In a text-property-based
implementation, on input they will be converted to text properties on
the characters controlled, and automatically converted back on
output.  Once again, those characters will neither be removed nor
added in the buffer.

 > > I see no reason why a text-property-based implementation should be
 > > lossy.
 > 
 > Because the user could type directional controls, and there's no way
 > for Emacs to know at all levels which one is to be treated in which
 > way.

What's wrong with reparsing the buffer from the beginning, treating
each change of value of the direction property as insertion of the
appropriate direction mark?  If there are redundant marks, of course
they would have to be indicated in some way.  But if that doesn't
work, I don't see how having explicit mark characters in the buffer
can work either.

 > > I don't understand.  If `put-text-property' and friends don't get that
 > > right already, more than bidi is in trouble, I should think.  What's
 > > special about bidi?
 > 
 > What is special is the fact that bidi needs nested regions with
 > different values for the same property.  Normally, if you put a
 > property with a value on a portion of text that has another value for
 > that property, the new value replaces the old one.

Sure.  But this is Lisp.  There's nothing that says that you are
limited to something as simple as 'ltr vs. 'rtl as the property value.
You could have a rather complex property, eg, containing the level of
the embedding as well as the resolved direction.  Or you could simply
replace the directional marks with a string on the preceding non-mark
character containing the mark characters that were present in the
source.  (Of course you'd need some hack for marks at the beginning of
buffer.)

It might not be worth it.  But then again...

Re: bidi-display-reordering is now non-nil by default

[Stephen J. Turnbull]

Stephen J. Turnbull writes:

 > Sure.  But this is Lisp.  There's nothing that says that you are
 > limited to something as simple as 'ltr vs. 'rtl as the property value.
 > You could have a rather complex property, eg, containing the level of
 > the embedding as well as the resolved direction.  Or you could simply
 > replace the directional marks with a string on the preceding non-mark
 > character containing the mark characters that were present in the
 > source.  (Of course you'd need some hack for marks at the beginning of
 > buffer.)
 > 
 > It might not be worth it.  But then again...

I should add that this is in the spirit of Stefan's "let's encourage
people to experiment in their own branches or after the release".
Most users with a need for bidi should be perfectly satisfied with the
current implementation.

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Cc: larsi@gnus.org,
>     list-general@mohsen.1.banan.byname.net,
>     emacs-devel@gnu.org
> Date: Fri, 05 Aug 2011 01:55:01 +0900
> 
> Eli Zaretskii writes:
> 
>  > Let's stick to the issue at hand: do you consider it a good idea to
>  > remove or add these characters in an otherwise unmodified buffer?
> 
> Er, you're the one who keep bringing up random irrelevancies like
> byte-level equality.

They are not irrelevant.  What you suggest runs the risk of adding or
removing LRM/RLM characters to/from a file against user expectations.

> And your question makes no sense.  In your implementation, they will
> be present as characters in the buffer, and they should neither be
> removed nor added per the Unicode standard.  In a text-property-based
> implementation, on input they will be converted to text properties on
> the characters controlled, and automatically converted back on
> output.  Once again, those characters will neither be removed nor
> added in the buffer.

Again, what if the user inserts another LRM?  In some positions, the
LRM does not change the directionality of the surrounding text, so
your text properties will be identical with or without it.  Then on
output to disk, this LRM will be lost.

>  > > I see no reason why a text-property-based implementation should be
>  > > lossy.
>  > 
>  > Because the user could type directional controls, and there's no way
>  > for Emacs to know at all levels which one is to be treated in which
>  > way.
> 
> What's wrong with reparsing the buffer from the beginning, treating
> each change of value of the direction property as insertion of the
> appropriate direction mark?

Reparsing the whole buffer upon each insertion?  Is that the way to
make redisplay fast and efficient?

> If there are redundant marks, of course they would have to be
> indicated in some way.

How do you indicate them, exactly?  Emacs has no features, except
again text properties, to indicate something like that.  In any case,
isn't it beginning to sound more and more complicated?

> But if that doesn't work, I don't see how having explicit mark
> characters in the buffer can work either.

Explicit marks work because the reordering algorithm does TRT with
them, whether they are redundant or not.  It doesn't care.  By not
caring it makes it very easy to preserve the byte stream and not risk
changing it behind user's back.

These are exactly the considerations that convinced me long ago that
leaving the explicit marks is the only reasonably safe and
uncomplicated way of doing this.

>  > > I don't understand.  If `put-text-property' and friends don't get that
>  > > right already, more than bidi is in trouble, I should think.  What's
>  > > special about bidi?
>  > 
>  > What is special is the fact that bidi needs nested regions with
>  > different values for the same property.  Normally, if you put a
>  > property with a value on a portion of text that has another value for
>  > that property, the new value replaces the old one.
> 
> Sure.  But this is Lisp.  There's nothing that says that you are
> limited to something as simple as 'ltr vs. 'rtl as the property value.
> You could have a rather complex property, eg, containing the level of
> the embedding as well as the resolved direction.

The _value_ doesn't matter.  It's the property symbol that cannot be
the same in overlapping regions, unless the values are identical.

> Or you could simply replace the directional marks with a string on
> the preceding non-mark character containing the mark characters that
> were present in the source.

And then move that string when text is inserted after the preceding
non-mark character, or that character is deleted, yes?  Sounds like
fun.

This horse has been beaten to death years ago, and it is still dead,
believe me.  Perhaps someone brilliant could come up with some
elaborate scheme which would somehow solve all these difficulties and
plug all the holes, in theory.  But we have such a simple and natural
alternative that it simply makes no good engineering sense whatsoever
to try to go this way, even if one hopes (and I don't) they will find
a bulletproof solution.

Using explicit marks does have its drawbacks, but they are minor and
mostly just need to get used to.  I'm the last one to disregard
usability considerations, but I'm quite sure users won't be anywhere
near the annoyance that Lars and you are afraid of.

In fact, I have already an experiment under way: one of the Emacs
modes already inserts explicit directional characters to tidy the
display; let's see when the first user complaint about that will
arrive.

Re: bidi-display-reordering is now non-nil by default

[Stephen J. Turnbull]

Eli Zaretskii writes:

 > They are not irrelevant.  What you suggest runs the risk of adding or
 > removing LRM/RLM characters to/from a file against user
 > expectations.

Sure, but byte-level equality is not part of that; character-level
equality is.

 > Again, what if the user inserts another LRM?

Insert another non-character "marker" in the buffer, using whichever
non-character strategy were using.

 > > What's wrong with reparsing the buffer from the beginning, treating
 > > each change of value of the direction property as insertion of the
 > > appropriate direction mark?
 > 
 > Reparsing the whole buffer upon each insertion?  Is that the way to
 > make redisplay fast and efficient?

No, that's a proof that it's *possible*, where your words claim it's
*im*possible.  Making it fast is a SMOP.  You say it's beyond you, and
that probably means it's beyond anybody competent enough in bidi to do
the implementation.  But let's not discourage anyone from trying. ;-)

 > How do you indicate them, exactly?  Emacs has no features, except
 > again text properties, to indicate something like that.  In any case,
 > isn't it beginning to sound more and more complicated?

Sure.  And the presence of non-graphic characters in the buffer is
going to make other code more complicated.  The question is, which
complexity is preferable?  You've made your choice, and Emacs has a
bidi implementation.  That's good, very good.  Nevertheless, I am
going to reserve judgment.

 > > But if that doesn't work, I don't see how having explicit mark
 > > characters in the buffer can work either.
 > 
 > Explicit marks work because the reordering algorithm does TRT with
 > them, whether they are redundant or not.  It doesn't care.  By not
 > caring it makes it very easy to preserve the byte stream and not risk
 > changing it behind user's back.

The algorithm will be the same, except that it needs to work with a
"virtual" stream where some characters are not present in the buffer.
This is no different from handling faces, which *could* be represented
as characters in the buffer (and *are* in HTML, for example -- which
of course has been deprecated in favor of CSS!  Hmm... :-).

The necessary buffering is a relatively small amount of complexity
compared to the bidi algorithm itself.

 > The _value_ doesn't matter.  It's the property symbol that cannot be
 > the same in overlapping regions, unless the values are identical.

Of course the value matters.  A 'direction property with a sequence
value can encode the whole stack, up to 61 levels.  Again, I wouldn't
want to maintain that design (space-inefficiency and the question of
consistency of neighboring regions are killers, I think), but there
are surely lighter-weight, more efficient designs.

 > > Or you could simply replace the directional marks with a string on
 > > the preceding non-mark character containing the mark characters that
 > > were present in the source.
 > 
 > And then move that string when text is inserted after the preceding
 > non-mark character, or that character is deleted, yes?  Sounds like
 > fun.

Put that way, not at all.  But you know what?  Emacs has long ago
solved such problems, at least most of them.  IIUC, in XEmacs, this
could easily be implemented with a zero-length extent with appropriate
stickiness attributes.  If Emacs doesn't already have such a device,
it would be easy enough to add a marker-with-direction extension by
maintaining a hashtable with keys markers and values mark characters.
Not terribly efficient, of course, but proof of concept.

 > Using explicit marks does have its drawbacks, but they are minor and
 > mostly just need to get used to.

There is way too much about Emacs that users (and developers!) "mostly
just need to get used to." :-(  Whether we can do much about it, I
don't know.  But I'm not going to give up yet. :-)

Thank you very much for taking the time out to explain your reasons
for your design choices.  I have a much better grasp of the practical
issues involved in implementing bidi in Emacsen now.

Re: bidi-display-reordering is now non-nil by default

[David Kastrup]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

>  > And then move that string when text is inserted after the preceding
>  > non-mark character, or that character is deleted, yes?  Sounds like
>  > fun.
>
> Put that way, not at all.  But you know what?  Emacs has long ago
> solved such problems, at least most of them.  IIUC, in XEmacs, this
> could easily be implemented with a zero-length extent with appropriate
> stickiness attributes.  If Emacs doesn't already have such a device,

Text properties can't be zero-length, overlays don't copy along with
text.  If this were different, you could copy an empty region and could
not be sure that it did not contain extents after all.

-- 
David Kastrup

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: "Stephen J. Turnbull" <stephen@xemacs.org>
> Cc: larsi@gnus.org,
>     list-general@mohsen.1.banan.byname.net,
>     emacs-devel@gnu.org
> Date: Fri, 05 Aug 2011 12:38:21 +0900
> 
> Eli Zaretskii writes:
> 
>  > They are not irrelevant.  What you suggest runs the risk of adding or
>  > removing LRM/RLM characters to/from a file against user
>  > expectations.
> 
> Sure, but byte-level equality is not part of that; character-level
> equality is.

LRM is also a character, for this purpose, yes?

>  > Again, what if the user inserts another LRM?
> 
> Insert another non-character "marker" in the buffer, using whichever
> non-character strategy were using.

And now what happens if the user wants to search for that LRM
character she just inserted?

>  > > What's wrong with reparsing the buffer from the beginning, treating
>  > > each change of value of the direction property as insertion of the
>  > > appropriate direction mark?
>  > 
>  > Reparsing the whole buffer upon each insertion?  Is that the way to
>  > make redisplay fast and efficient?
> 
> No, that's a proof that it's *possible*, where your words claim it's
> *im*possible.

Impossible or unacceptable -- is there really a difference in
practice?

> Making it fast is a SMOP.  You say it's beyond you, and
> that probably means it's beyond anybody competent enough in bidi to do
> the implementation.  But let's not discourage anyone from trying. ;-)

There's a saying that smart people learn from their experience, but
wise people learn from that of others.  If someone is wise and wants
to learn from my experience, please read the history and the diffs of
bug#9218.  It had to do with a flawed design of certain aspects of
bidi iteration whereby sometimes the display engine had to look from
some point in the buffer to its very end.  The result was a completely
unusable Emacs in the buffers that were hitting this design flaw
(e.g., Org Mode buffers of a few MB size).

>  > How do you indicate them, exactly?  Emacs has no features, except
>  > again text properties, to indicate something like that.  In any case,
>  > isn't it beginning to sound more and more complicated?
> 
> Sure.  And the presence of non-graphic characters in the buffer is
> going to make other code more complicated.

Again, LRM is just a character, like ZWNJ and friends.  We need to
support such characters in files anyway.  And we already started, with
the glyphless-char-display feature.

>  > > But if that doesn't work, I don't see how having explicit mark
>  > > characters in the buffer can work either.
>  > 
>  > Explicit marks work because the reordering algorithm does TRT with
>  > them, whether they are redundant or not.  It doesn't care.  By not
>  > caring it makes it very easy to preserve the byte stream and not risk
>  > changing it behind user's back.
> 
> The algorithm will be the same, except that it needs to work with a
> "virtual" stream where some characters are not present in the buffer.
> This is no different from handling faces, which *could* be represented
> as characters in the buffer (and *are* in HTML, for example -- which
> of course has been deprecated in favor of CSS!  Hmm... :-).

Actually, I think dealing with "virtual" characters means at least
lots of changes in Emacs if not larger trouble.  Up to v24, Emacs
assumed that the correspondence between buffer text and text on
display is mostly 1:1.  Sure, display strings, invisible text,
variable fonts, and other display features break that to some extent,
but by and large, this was true.  Emacs 24 changes that some more due
to support of bidi.  But bidi support is _a_display_only_feature_, and
the current design sticks to that almost religiously.  Again, the need
to insert LRM/RLM etc. here and there violates the "display-only"
thing, but one could claim that this is unrelated to bidi display per
se: if we don't care about good looks in specially formatted buffers,
we can disregard this issue; the display will still be "correct" per
the UBA.

This assumption, of the basic 1:1 correspondence between buffer text
and the display, is very fundamental and affects many Emacs features
not directly related to display.  One such set of features is column
counting and the vertical scrolling and indentation features that are
based on it.  If you look under the hood, you will see that some/many
of the functions involved in the implementation of this walk buffer
text, not the display structures.  (Being dependent on display
structures means that the related features cannot work if the display
is not up to date, which is unacceptable.)  Any idea whose result is
"virtual" characters not in the buffer means a tremendous complication
in these features, for reasons that I hope are obvious.  In a
nutshell, a display-only feature will leak all over the code that
works with buffer text.  I won't argue whether this is impractical or
impossible, but I hope you will at least agree that it's undesirable.

>  > The _value_ doesn't matter.  It's the property symbol that cannot be
>  > the same in overlapping regions, unless the values are identical.
> 
> Of course the value matters.  A 'direction property with a sequence
> value can encode the whole stack, up to 61 levels.

Then you'd need to change this value on every edit of the related
text.

> Again, I wouldn't want to maintain that design (space-inefficiency
> and the question of consistency of neighboring regions are killers,
> I think), but there are surely lighter-weight, more efficient
> designs.

I doubt the "surely" part.

> IIUC, in XEmacs, this could easily be implemented with a zero-length
> extent with appropriate stickiness attributes.

I Only know about Emacs stickiness.  With that, this idea will lead to
proliferation of characters with the "mark" value, as text around it
is added/deleted.  You will need to work hard to maintain that so that
there's only one place with that value.

> Thank you very much for taking the time out to explain your reasons
> for your design choices.  I have a much better grasp of the practical
> issues involved in implementing bidi in Emacsen now.

You are welcome.

Re: bidi-display-reordering is now non-nil by default

[Stephen J. Turnbull]

Eli Zaretskii writes:

 > > Sure, but byte-level equality is not part of that; character-level
 > > equality is.
 > 
 > LRM is also a character, for this purpose, yes?

Of course.  What makes you think I might deny that?

 > > No, that's a proof that it's *possible*, where your words claim it's
 > > *im*possible.
 > 
 > Impossible or unacceptable -- is there really a difference in
 > practice?

Yes.  You can't optimize impossible.  Optimizing unacceptable is what
puts programmers' kids through college.

 > > Sure.  And the presence of non-graphic characters in the buffer is
 > > going to make other code more complicated.
 > 
 > Again, LRM is just a character, like ZWNJ and friends.  We need to
 > support such characters in files anyway.  And we already started, with
 > the glyphless-char-display feature.

Support in files and treating them as characters in buffers are two
different issues.  Mule already requires supporting lots of characters
in files that never appear in buffers.  Unicode helps with that, but
it has its own problems.

 > I Only know about Emacs stickiness.

Yeah, I'm beginning to get the impression that "impossible" is a word
that may be appropriately applied to Emacs in this area. ;-)  Oh, well.

Re: bidi-display-reordering is now non-nil by default

[Stefan Monnier]

Regarding using LRM or text-properties for Gnus summary fields:

Maybe we can do better, but we're in feature freeze so anything that
needs a new feature from the bidi code is just not an option for 24.1.
There are enough other problems in the current bidi support to keep
Eli busy.

This said, I encourage other people to try and add some alternative
solution (such as adding a text property that can override the
bidi-class of the characters it covers, so you can have a "segment
separator" which is not a TAB), to see how well it works.  That will
also hopefully bring more eyes on the bidi code, which can't hurt.


        Stefan

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Thu, 04 Aug 2011 09:59:26 -0400
> 
> This said, I encourage other people to try and add some alternative
> solution (such as adding a text property that can override the
> bidi-class of the characters it covers, so you can have a "segment
> separator" which is not a TAB), to see how well it works.

As explained elsewhere, this will need a significant change in
low-level scanning routines within bidi.c.  Currently, they are mostly
oblivious to text properties.

Re: bidi-display-reordering is now non-nil by default

[Michael Welsh Duggan]

Eli Zaretskii <eliz@gnu.org> writes:

> You are wrong.  LRM, RLM, etc. exist _precisely_ for these purposes:
> to arrange bidirectional text for display as the display designer
> wishes, where the implicit reordering mandated by the UBA does not
> give satisfactory results.  These _explicit_ directional control
> characters exist for when the _implicit_ reordering fails.  It's the
> functional equivalent of the corresponding HTML directives -- you
> won't say that HTML is "suboptimal" when it dictates to the browser
> how to render bidirectional text, would you?

[...]

> This issue was discussed at length long ago, when the basic design of
> bidi support for Emacs was on the table.  Text properties were
> suggested almost immediately, then rejected after prolonged debates,
> because they simply aren't up to this job, and because the Unicode
> Standard already tells us how to DTRT, we just need to heed.

I have no issue with using directional control characters in the buffer
to handle alignment issues, etc.  I do have the following questions,
though, and am looking for suggestions for how they should be
implemented.

Scenario: Say we are desirous of updating XML mode to protect the
neutral characters that make up tags so they don't end up in visually
confusing locations.  This can be done by adding proper directional
codes in the right locations.  However, we would need to solve the
following two questions:

1) We need to strip these directional characters (but not other ones in
the data sections of the XML) when saving the files.

2) We don't really want cursor movement to stop on each of these
directional characters.  They are there to make display look correct,
but are not part of the data stream that the user is editing.

3) When doing search, people are not going to want to have to add the
directional codes to their search strings in order to, say, search for a
tag next to some text.  These formatting characters need to be
"invisible" with respect to isearch and friends.

4) Not as important, but in this case we would want the formatting
characters we add in order to make markup display appropriately to be
non-visible, whereas formatting characters in the user text, we might
want displayed using thin-space or as a boxed character.

How should these problems be solved?  Or what feature(s) should be added
to allow this to be implemented?

-- 
Michael Welsh Duggan
(md5i@md5i.com)

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Michael Welsh Duggan <md5i@md5i.com>
> Cc: Lars Magne Ingebrigtsen <larsi@gnus.org>,  list-general@mohsen.1.banan.byname.net,  emacs-devel@gnu.org
> Date: Thu, 04 Aug 2011 23:41:27 -0400
> 
> Scenario: Say we are desirous of updating XML mode to protect the
> neutral characters that make up tags so they don't end up in visually
> confusing locations.  This can be done by adding proper directional
> codes in the right locations.

I was not arguing for inserting directional control characters into a
buffer that visits a file.  If there are no such directional control
characters in the file, it is a very bad idea for Emacs to insert them
for display purposes: it will run a high risk of leaking those extra
characters back to the file.  Your questions all explain why this
would be a bad idea.

The current features of bidirectional display do not yet support
display of structured files, such as XML or comments and strings in
program files (the latter "mostly" works, but only by accident).
Supporting XML and similar structured text will need additional
features that are not yet there.  I don't even have a clear idea yet
of what such a features would be.  Sorry.

The only thing you can do with the current bidi support that should
work is to cover each string that may need reordering with a display
property whose value is that same string.  If you carefully exclude
from the string all the neutral characters that get in the way of the
"normal" reordering, you should get the effect that you expect.  (I
think; I didn't actually try that.)

The disadvantage of this is that _editing_ those strings will be
complicated: you will need to intercept cursor motion command and use
the `cursor' property a lot to let the user think the cursor can enter
such a string.  You will also need to recompute the display string
each time its origin in the buffer is modified.  That's tedious at
best, but at least it has a chance to work well with the current
infrastructure.  I think.

If you try this suggestion, and find it feasible, then perhaps it
could be a basis for solving this problem, or even a large part of
the solution.

Re: bidi-display-reordering is now non-nil by default

[Chong Yidong]

Eli Zaretskii <eliz@gnu.org> writes:

>> Please correct me if this isn't what all y'all have in mind.
>
> Using LRM/RLM _is_ TRT, Lars.  It may sound weird to you, but it's
> like democracy: the worst political system, except all the rest.  I
> wrote the entire Hebrew tutorial with these characters, and didn't see
> any problems.  It just takes some time to get used to it, that's all.
> R2L users already are used to it.

If bidi-display-reordering is nil in the Gnus summary buffer, is it
still necessary to add these LRM/RLM characters?

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Chong Yidong <cyd@stupidchicken.com>
> Cc: Lars Magne Ingebrigtsen <larsi@gnus.org>,
>         list-general@mohsen.1.banan.byname.net, emacs-devel@gnu.org
> Date: Fri, 05 Aug 2011 13:56:36 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Please correct me if this isn't what all y'all have in mind.
> >
> > Using LRM/RLM _is_ TRT, Lars.  It may sound weird to you, but it's
> > like democracy: the worst political system, except all the rest.  I
> > wrote the entire Hebrew tutorial with these characters, and didn't see
> > any problems.  It just takes some time to get used to it, that's all.
> > R2L users already are used to it.
> 
> If bidi-display-reordering is nil in the Gnus summary buffer, is it
> still necessary to add these LRM/RLM characters?

No.  But then R2L text in Subject and From fields will not be
displayed correctly (not reordered as users would expect).

Re: bidi-display-reordering is now non-nil by default

[Chong Yidong]

Eli Zaretskii <eliz@gnu.org> writes:

>> If bidi-display-reordering is nil in the Gnus summary buffer, is it
>> still necessary to add these LRM/RLM characters?
>
> No.  But then R2L text in Subject and From fields will not be
> displayed correctly (not reordered as users would expect).

Sorry, I misspoke.  What I meant to ask is: if bidi-paragraph-direction
is set to left-to-right in the Gnus summary buffer, is it still
necessary to add these LRM/RLM characters?

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Chong Yidong <cyd@stupidchicken.com>
> Cc: larsi@gnus.org, list-general@mohsen.1.banan.byname.net,
>         emacs-devel@gnu.org
> Date: Fri, 05 Aug 2011 14:45:54 -0400
> 
> if bidi-paragraph-direction is set to left-to-right in the Gnus
> summary buffer, is it still necessary to add these LRM/RLM
> characters?

Yes.  This issue has nothing to do with paragraph direction (and the
Gnus summary buffer ought to set bidi-paragraph-direction to
left-to-right anyway).  This issue is about reordering a line of text
that is not really a contiguous text, but rather a series of separate
fields.  Either each field needs to be reordered separately, which can
be done by separating them with TABs, or we need to "fix" the default
reordering (which assumes the whole line is a single piece of text) by
inserting LRM characters in strategical places.

Re: bidi-display-reordering is now non-nil by default

[Chong Yidong]

Eli Zaretskii <eliz@gnu.org> writes:

> Yes.  This issue has nothing to do with paragraph direction (and the
> Gnus summary buffer ought to set bidi-paragraph-direction to
> left-to-right anyway).  This issue is about reordering a line of text
> that is not really a contiguous text, but rather a series of separate
> fields.  Either each field needs to be reordered separately, which can
> be done by separating them with TABs, or we need to "fix" the default
> reordering (which assumes the whole line is a single piece of text) by
> inserting LRM characters in strategical places.

Thanks, I think I see.

Still, it seems better not to change Gnus to proactively insert LRM
characters, but leave it to those users who care to customize it as
necessary.  In the default value of gnus-summary-line-format,

  "%U%R%z%I%(%[%4L: %-23,23f%]%) %s\n"

the subject (%s) is followed by a newline.  If a user wants to change
this to include, say, the article number, that could be done via

  "%U%R%z%I%(%[%4L: %-23,23f%]%) %s\u200E%N\n"

This has the advantage of avoiding inserting the insertion of excess LRM
characters by default.  The disadvantage, of course, is that it requires
some extra knowledge from users, but we can handle this by adding a note
to the docstring.

Re: bidi-display-reordering is now non-nil by default

[Jason Rumney]

Chong Yidong <cyd@stupidchicken.com> writes:

> Still, it seems better not to change Gnus to proactively insert LRM
> characters, but leave it to those users who care to customize it as
> necessary.

Users who get a lot of RTL email may be motivated enough to do that, but
users who seldom get such mail (and what they do get is probably mostly
spam) will still see the muddled up summary line as a bug.

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Chong Yidong <cyd@stupidchicken.com>
> Cc: larsi@gnus.org, list-general@mohsen.1.banan.byname.net,
>         emacs-devel@gnu.org
> Date: Fri, 05 Aug 2011 17:54:54 -0400
> 
> Still, it seems better not to change Gnus to proactively insert LRM
> characters, but leave it to those users who care to customize it as
> necessary.

I don't think this is a good idea.  It requires users to know too much
about the reordering algorithm.  From this discussion you may conclude
that those who need it will have that knowledge anyway, but I think
this conclusion would be wrong: the participants of this discussion
who argue for usage of these characters are not typical users.

> This has the advantage of avoiding inserting the insertion of excess LRM
> characters by default.
> 
> The disadvantage, of course, is that it requires some extra
> knowledge from users, but we can handle this by adding a note to the
> docstring.

We have no hope of giving precise instructions about this in a doc
string.  All we can do is mention those characters.  Users who don't
know about the UBA will have hard time finding where to insert them by
trial and error.  I can tell you from my experience of writing
TUTORIAL.he that it took _me_ a few trials and errors to know which
directional control to insert in some cases.

Anyway, I don't understand the opposition to using directional
controls, here and elsewhere.  We already use special characters, like
spaces and '[..]' above, to make the summary line more visually
appealing.  Why are LRM and RLM different?  They are just characters,
and they are barely visible.  If even the current thin-space
appearance is deemed too visible, we could change the default value of
glyphless-char-display to make them not visible at all.  (I think
Stefan said we should do that, see the discussion referenced below).
On top of that, summary buffers are rarely traversed by users, so the
chances they will pay attention are slim at best.  So what is the
reason(s) for such opposition?  I just don't get it.

Btw, a similar discussion about buffer names that include R2L
characters ended up concluding that we should use RLM/LRM controls to
make FOO<2> etc. display correctly in all kinds of context, see the
thread that started here:

  https://lists.gnu.org/archive/html/emacs-devel/2011-06/msg00712.html

In that thread, _I_ had my reservations, and others, including Stefan,
convinced me that using directional controls is TRT.  How is this case
different?

Re: bidi-display-reordering is now non-nil by default

[Chong Yidong]

Eli Zaretskii <eliz@gnu.org> writes:

>> Still, it seems better not to change Gnus to proactively insert LRM
>> characters, but leave it to those users who care to customize it as
>> necessary.
>
> I don't think this is a good idea.  It requires users to know too much
> about the reordering algorithm.  From this discussion you may conclude
> that those who need it will have that knowledge anyway, but I think
> this conclusion would be wrong: the participants of this discussion
> who argue for usage of these characters are not typical users.

OK, how about introducing a function similar to the following?

(defun string-mark-left-to-right (str)
  "Return a string based on STR that can be embedded in LTR text.
If STR ends in right-to-left (RTL) script, return a string
consisting of STR followed by a left-to-right mark (LRM)
character.  Otherwise, return STR."
  (if --??--this-string-needs-an-lrm--??--
      (concat str "\u200E")
    str))

(Can you tell us how to fill in the missing piece above?)

*** lisp/gnus/gnus-sum.el	2011-08-05 23:29:17 +0000
--- lisp/gnus/gnus-sum.el	2011-08-07 17:20:07 +0000
***************
*** 5357,5362 ****
--- 5357,5367 ----
  		   gnus-summary-same-subject
  		 subject))
  	      (t gnus-summary-same-subject)))
+ 
+ 	    (if gnus-tmp-subject-or-nil
+ 		(setq gnus-tmp-subject-or-nil
+ 		      (string-mark-left-to-right gnus-tmp-subject-or-nil)))
+ 
  	    (if (and (eq gnus-summary-make-false-root 'adopt)
  		     (= gnus-tmp-level 1)
  		     (memq number gnus-tmp-gathered))

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Chong Yidong <cyd@stupidchicken.com>
> Cc: larsi@gnus.org, list-general@mohsen.1.banan.byname.net,
>         emacs-devel@gnu.org
> Date: Sun, 07 Aug 2011 13:21:41 -0400
> 
> (defun string-mark-left-to-right (str)
>   "Return a string based on STR that can be embedded in LTR text.
> If STR ends in right-to-left (RTL) script, return a string
> consisting of STR followed by a left-to-right mark (LRM)
> character.  Otherwise, return STR."
>   (if --??--this-string-needs-an-lrm--??--
>       (concat str "\u200E")
>     str))
> 
> (Can you tell us how to fill in the missing piece above?)

How about

  (if t

?

Seriously, though: why do you need any conditions here?  An LRM will
have no effect if str is not reordered, so why not put it there
unconditionally?

But if you insist on doing that only when str might need that, then I
think you want to check each character in str with
get-char-code-property, like this:

  (get-char-code-property (aref str i) 'bidi-class)

If this returns 'R for any character, it means the string _might_ need
LRM to be appended.

Re: bidi-display-reordering is now non-nil by default

[Chong Yidong]

Eli Zaretskii <eliz@gnu.org> writes:

> Seriously, though: why do you need any conditions here?  An LRM will
> have no effect if str is not reordered, so why not put it there
> unconditionally?

Suppose a user kills some text from a Gnus summary buffer which contains
these LRM characters, and yanks it into a latin-1 file.  Suddenly, that
file can't be saved, even though there is no non-latin-1 script in the
buffer clearly visible.

If we can trivially avoid most such occurrences, why not do so?

> But if you insist on doing that only when str might need that, then I
> think you want to check each character in str with
> get-char-code-property, like this:
>
>   (get-char-code-property (aref str i) 'bidi-class)
>
> If this returns 'R for any character, it means the string _might_ need
> LRM to be appended.

OK, let's go with this unless someone comes up with a better idea.

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Chong Yidong <cyd@stupidchicken.com>
> Cc: larsi@gnus.org, list-general@mohsen.1.banan.byname.net,
>         emacs-devel@gnu.org
> Date: Tue, 09 Aug 2011 12:07:05 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Seriously, though: why do you need any conditions here?  An LRM will
> > have no effect if str is not reordered, so why not put it there
> > unconditionally?
> 
> Suppose a user kills some text from a Gnus summary buffer which contains
> these LRM characters, and yanks it into a latin-1 file.  Suddenly, that
> file can't be saved, even though there is no non-latin-1 script in the
> buffer clearly visible.

When you type "C-x C-s", Emacs clearly shows the characters that
cannot be encoded as Latin-1.  We all bump into such issues from time
to time, especially when communicating via email or news articles.  I
see no particular problem here.

> >   (get-char-code-property (aref str i) 'bidi-class)
> >
> > If this returns 'R for any character, it means the string _might_ need
> > LRM to be appended.
> 
> OK, let's go with this unless someone comes up with a better idea.

I think separating fields by TABs covered by `invisible' text
properties might be an alternative worth considering.

Re: bidi-display-reordering is now non-nil by default

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Chong Yidong <cyd@stupidchicken.com>
>> Cc: larsi@gnus.org, list-general@mohsen.1.banan.byname.net,
>>         emacs-devel@gnu.org
>> Date: Tue, 09 Aug 2011 12:07:05 -0400
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > Seriously, though: why do you need any conditions here?  An LRM will
>> > have no effect if str is not reordered, so why not put it there
>> > unconditionally?
>> 
>> Suppose a user kills some text from a Gnus summary buffer which contains
>> these LRM characters, and yanks it into a latin-1 file.  Suddenly, that
>> file can't be saved, even though there is no non-latin-1 script in the
>> buffer clearly visible.
>
> When you type "C-x C-s", Emacs clearly shows the characters that
> cannot be encoded as Latin-1.  We all bump into such issues from time
> to time, especially when communicating via email or news articles.  I
> see no particular problem here.

We see such issues in general only when there has been some external
source of foreign characters in play.  But when I am copying and pasting
elements from an ASCII-only user interface showing ASCII-only summary
lines, there has not been any external source of foreign characters
involved.  Emacs should not become unpredictable in an ASCII-only
environment with ASCII-only data sources.  Let alone in a latin-1
environment.

-- 
David Kastrup

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Tue, 09 Aug 2011 18:30:56 +0200
> 
> Emacs should not become unpredictable in an ASCII-only environment
> with ASCII-only data sources.  Let alone in a latin-1 environment.

Dealing with email has long ago stopped being an ASCII-only or
Latin-1-only environment.

Re: bidi-display-reordering is now non-nil by default

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>> Date: Tue, 09 Aug 2011 18:30:56 +0200
>> 
>> Emacs should not become unpredictable in an ASCII-only environment
>> with ASCII-only data sources.  Let alone in a latin-1 environment.
>
> Dealing with email has long ago stopped being an ASCII-only or
> Latin-1-only environment.

What about "ASCII-only data sources" did you not understand?  We are not
talking about the situation when a data source (like a mail) happens to
actually be bidi.

When I am replying to an ASCII-only mail, I don't want to have Unicode
direction marks pasted into its subject line or wherever else.  When I
cut and paste from a summary buffer of an English-only mailing list
without any foreign-language spam or senders, I don't want Unicode
direction marks in the result.  I have mm-coding-system-priorities set
to (iso-8859-1 iso-8859-15 utf-8) and I don't want it to switch without
good reason to a coding system that will unnecessarily limit my audience
to multibyte-capable readers.  This list would actually start with
ascii, except that I don't know any system that would display pure ASCII
articles any worse when they are declared as iso-8859-1.

-- 
David Kastrup

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Tue, 09 Aug 2011 19:26:18 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: David Kastrup <dak@gnu.org>
> >> Date: Tue, 09 Aug 2011 18:30:56 +0200
> >> 
> >> Emacs should not become unpredictable in an ASCII-only environment
> >> with ASCII-only data sources.  Let alone in a latin-1 environment.
> >
> > Dealing with email has long ago stopped being an ASCII-only or
> > Latin-1-only environment.
> 
> What about "ASCII-only data sources" did you not understand?

What part of "there are no ASCII-only data sources" did you not
understand?

> When I am replying to an ASCII-only mail, I don't want to have Unicode
> direction marks pasted into its subject line or wherever else.

No one said anything about replying.  Stay on the subject, if you want
a serious discussion.

> [...] English-only mailing list without any foreign-language spam or
> senders [...]

Where do you find such things nowadays?  They don't exist.

> limit my audience to multibyte-capable readers

What are multibyte-uncapable readers in use nowadays?  Wake up, it's
2011 out there!  The dumb MS Outlook supports directional marks for
quite some time.

Re: bidi-display-reordering is now non-nil by default

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>> Date: Tue, 09 Aug 2011 19:26:18 +0200
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> From: David Kastrup <dak@gnu.org>
>> >> Date: Tue, 09 Aug 2011 18:30:56 +0200
>> >> 
>> >> Emacs should not become unpredictable in an ASCII-only environment
>> >> with ASCII-only data sources.  Let alone in a latin-1 environment.
>> >
>> > Dealing with email has long ago stopped being an ASCII-only or
>> > Latin-1-only environment.
>> 
>> What about "ASCII-only data sources" did you not understand?
>
> What part of "there are no ASCII-only data sources" did you not
> understand?

You are confusing an antecedent with an absolute statement.  An absolute
statement that is just wrong.  For example, this mail is an ASCII-only
data source.

>> [...] English-only mailing list without any foreign-language spam or
>> senders [...]
>
> Where do you find such things nowadays?  They don't exist.

Reverting to sentence part surgery in order to let your point stand is
pretty much equivalent to admitting its untenability.

>> limit my audience to multibyte-capable readers
>
> What are multibyte-uncapable readers in use nowadays?  Wake up, it's
> 2011 out there!  The dumb MS Outlook supports directional marks for
> quite some time.

Typical Usenet groups I frequent not rarely use newsreaders like Xnews,
Forte Agent and other tools that have a lot of problems dealing with
multibyte characters, in particular when replying to mail not matching
the local environment.

If you want to wake up to 2011, Usenet is dead, and everybody uses web
forums.  But I am still communicating with a lot of dinosaurs.  People
who still, 2011 or not, tend to get annoyed at HTML mail.

I am contributing code to projects where I don't want to see utf-8
characters checked in that I have not explicitly typed myself, let alone
directional characters.

-- 
David Kastrup

Re: bidi-display-reordering is now non-nil by default

[Richard Stallman]

The FSF will not send HTML mail.


-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use free telephony http://directory.fsf.org/category/tel/

Re: bidi-display-reordering is now non-nil by default

[Stephen J. Turnbull]

Eli Zaretskii writes:

 > What part of "there are no ASCII-only data sources" did you not
 > understand?

That you could seriously post such a statement is what I don't understand.

Re: bidi-display-reordering is now non-nil by default

[Chong Yidong]

> When you type "C-x C-s", Emacs clearly shows the characters that
> cannot be encoded as Latin-1.

As you said yourself, users should not need to know about the details of
how bidi works.  So we can't assume that they know what left-to-right
marks are, so they can't know whether the characters that Emacs is
complaining about are important to keep.

Re: bidi-display-reordering is now non-nil by default

[Stefan Monnier]

>> When you type "C-x C-s", Emacs clearly shows the characters that
>> cannot be encoded as Latin-1.
> As you said yourself, users should not need to know about the details of
> how bidi works.  So we can't assume that they know what left-to-right
> marks are, so they can't know whether the characters that Emacs is
> complaining about are important to keep.

LRM marks aren't perfect, but I think that for Emacs-24.1, it'll have to
be "the best we have", otherwise it would delay the release too much.

Also, while I'm here.  I re-suggest to change the default of
bidi-paragraph-direction to left-to-right, since I think that preserves
backward compatibility a lot better.  We can set it to nil in
the major modes where we know it can be useful, such as text-mode.


        Stefan

Re: bidi-display-reordering is now non-nil by default

[David Kastrup]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> When you type "C-x C-s", Emacs clearly shows the characters that
>>> cannot be encoded as Latin-1.
>> As you said yourself, users should not need to know about the details of
>> how bidi works.  So we can't assume that they know what left-to-right
>> marks are, so they can't know whether the characters that Emacs is
>> complaining about are important to keep.
>
> LRM marks aren't perfect, but I think that for Emacs-24.1, it'll have to
> be "the best we have", otherwise it would delay the release too much.

Using LRM is ok, but Emacs should not insert LRM marks on its own in
pure left-to-right text.

I already suggested that `format' could be made responsible for
inserting LRM marks.  One possibility would be a special format field
that would send an appropriate mark when left- and righthand side of the
string had differing directionality.  Another possibility would be
completely self-organized: namely when otherwise formatting fields would
get split or reordered.

-- 
David Kastrup

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Wed, 10 Aug 2011 03:14:48 +0200
> 
> Emacs should not insert LRM marks on its own in pure left-to-right
> text.

Then what about the other alternative that was proposed: separate
fields with TABs, possibly invisible?  That would at most give you TAB
characters, which are just ASCII.  No one responded to that suggestion
yet.

Re: bidi-display-reordering is now non-nil by default

[Chong Yidong]

Eli Zaretskii <eliz@gnu.org> writes:

> Then what about the other alternative that was proposed: separate
> fields with TABs, possibly invisible?  That would at most give you TAB
> characters, which are just ASCII.  No one responded to that suggestion
> yet.

Then you get TABs popping up unexpectedly on cut and paste (since
invisible properties are discarded by yank-excluded-properties).

Conceptually, it makes sense for buffer text to contain LRMs for
handling embedded RTL text, so there is no reason for Emacs to avoid
inserting them---as long as RTL text is really present.

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Chong Yidong <cyd@stupidchicken.com>
> Cc: David Kastrup <dak@gnu.org>, emacs-devel@gnu.org
> Date: Wed, 10 Aug 2011 12:07:03 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Then what about the other alternative that was proposed: separate
> > fields with TABs, possibly invisible?  That would at most give you TAB
> > characters, which are just ASCII.  No one responded to that suggestion
> > yet.
> 
> Then you get TABs popping up unexpectedly on cut and paste (since
> invisible properties are discarded by yank-excluded-properties).

How is that different from using invisible properties in Info mode?
You cut/paste from a manual, and the hidden text pops up
"unexpectedly".

Re: bidi-display-reordering is now non-nil by default

[Chong Yidong]

Eli Zaretskii <eliz@gnu.org> writes:

> How is that different from using invisible properties in Info mode?
> You cut/paste from a manual, and the hidden text pops up
> "unexpectedly".

And that is not a good thing.  There's no reason to make the problem
worse if we can help it.

I haven't seen an argument against the idea of introducing a
`string-mark-left-to-right' function; is there any?

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Chong Yidong <cyd@stupidchicken.com>
> Cc: dak@gnu.org, emacs-devel@gnu.org
> Date: Wed, 10 Aug 2011 12:52:24 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > How is that different from using invisible properties in Info mode?
> > You cut/paste from a manual, and the hidden text pops up
> > "unexpectedly".
> 
> And that is not a good thing.  There's no reason to make the problem
> worse if we can help it.
> 
> I haven't seen an argument against the idea of introducing a
> `string-mark-left-to-right' function; is there any?

You won't hear it from me.

Re: bidi-display-reordering is now non-nil by default

[Mohsen BANAN]

>>>>> On Wed, 10 Aug 2011 12:52:24 -0400, Chong Yidong <cyd@stupidchicken.com> said:

  Chong> Eli Zaretskii <eliz@gnu.org> writes:
  >> How is that different from using invisible properties in Info mode?
  >> You cut/paste from a manual, and the hidden text pops up
  >> "unexpectedly".

  Chong> And that is not a good thing.  There's no reason to make the problem
  Chong> worse if we can help it.

  Chong> I haven't seen an argument against the idea of introducing a
  Chong> `string-mark-left-to-right' function; is there any?

I am all for it.

Permit me to make a few additional very minor
suggestions that build on what you suggested.

 - Let's add a bidi- prefix to these function
   names.

 - Let's have two functions instead of one. 
   The one with -perhaps- does the if, the one 
   without does it always.

 - Let's put these is a file like bidi-kit.el
   somewhere on the trunk soon. So we can use them
   right away. It could be right next to
   bidi-menu.el starting point that suggested
   previously.

Note that when the -perhaps- version is used,
non-RTL script (e.g., ascii), remains pure ...

Thanks,

...Mohsen


--- starting point for bidi-kit.el starts ---


(defun bidi-string-perhaps-mark-left-to-right (str)
  "Return a string based on STR that can be embedded in LTR text.
If STR ends in right-to-left (RTL) script, return a string
consisting of STR followed by a left-to-right mark (LRM)
character.  Otherwise, return STR."
  (if --??--this-string-needs-an-lrm--??--
    ...(get-char-code-property (aref str i)  'bidi-class)... Check for R...
      (concat str "\u200E")
    str))


(defun bidi-string-mark-left-to-right (str)
  "Return a string based on STR that can be embedded in LTR text.
Return a string
consisting of STR followed by a left-to-right mark (LRM)
character. 
       (concat str "\u200E"))


--- starting point for bidi-kit.el ends ---

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Tue, 09 Aug 2011 21:03:14 -0400
> 
> Also, while I'm here.  I re-suggest to change the default of
> bidi-paragraph-direction to left-to-right, since I think that preserves
> backward compatibility a lot better.

You never explained what backward compatibility is alluded to here.

Anyway, that suggestion was not forgotten, I just want to wait with it
a bit more, to see if any additional problems are spotted with the
dynamic paragraph direction.

> We can set it to nil in the major modes where we know it can be
> useful, such as text-mode.

What other modes will need this?

Also, do we care about the case that the user sets the default value
to nil?

Re: bidi-display-reordering is now non-nil by default

[Stefan Monnier]

>> Also, while I'm here.  I re-suggest to change the default of
>> bidi-paragraph-direction to left-to-right, since I think that preserves
>> backward compatibility a lot better.
> You never explained what backward compatibility is alluded to here.

Emacs has done only ever left-to-right until now.  So if we use
right-to-left where left-to-right was needed, it's a regression, whereas
if we use  left-to-right where right-to-left was needed it's not.

> Anyway, that suggestion was not forgotten, I just want to wait with it
> a bit more, to see if any additional problems are spotted with the
> dynamic paragraph direction.

OK.

>> We can set it to nil in the major modes where we know it can be
>> useful, such as text-mode.
> What other modes will need this?

No idea.

> Also, do we care about the case that the user sets the default value
> to nil?

Not really, but I think it's good to keep the current explicit settings
to `left-to-right' in things like prog-mode.  It's also OK to add such
settings even if the default is already left-to-right.


        Stefan

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Wed, 10 Aug 2011 09:20:34 -0400
> 
> >> Also, while I'm here.  I re-suggest to change the default of
> >> bidi-paragraph-direction to left-to-right, since I think that preserves
> >> backward compatibility a lot better.
> > You never explained what backward compatibility is alluded to here.
> 
> Emacs has done only ever left-to-right until now.  So if we use
> right-to-left where left-to-right was needed, it's a regression, whereas
> if we use  left-to-right where right-to-left was needed it's not.

You are judging this from the POV or a non-R2L reader, who doesn't
really care about correct display of R2L text.  From the POV of R2L
readers, what Emacs did before v24 is a "regression", and the current
behavior is the correct one.

It is true that R2L text will be readable in Emacs 24 even with
left-to-right paragraphs, whereas in previous versions it was
illegible, but it still looks wrong to the R2L eye, and in some cases
(like very long lines that mix R2L and L2R) it's downright annoying.

It would be a real regression if Emacs would sometimes display strict
L2R text as right-to-left paragraphs.  But this is not the case.

IOW, Emacs 24 explicitly breaks "backward compatibility" in this
regard, and for a very good reason.  Some applications need to adapt
to that, true, but this adaptation is the remaining 10% of the long
journey from 100% non-support of bidi to 100% support.

> >> We can set it to nil in the major modes where we know it can be
> >> useful, such as text-mode.
> > What other modes will need this?
> 
> No idea.

Well, then how do we make sure that changing the default to
left-to-right will not make things worse than leaving it at nil?  The
list of modes that need left-to-right paragraphs and do not yet get
them due to inheritance from prog-mode is known (and currently very
short), because people complained.  Relying on similar complaints
after changing the default to left-to-right is not the best idea,
since the number of R2L users is much smaller than that of the other
kind.

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00 autolearn=ham
> 	version=3.3.1
> From: Chong Yidong <cyd@stupidchicken.com>
> Cc: larsi@gnus.org, list-general@mohsen.1.banan.byname.net,
>         emacs-devel@gnu.org
> Date: Tue, 09 Aug 2011 18:26:34 -0400
> 
> > When you type "C-x C-s", Emacs clearly shows the characters that
> > cannot be encoded as Latin-1.
> 
> As you said yourself, users should not need to know about the details of
> how bidi works.  So we can't assume that they know what left-to-right
> marks are, so they can't know whether the characters that Emacs is
> complaining about are important to keep.

I sometimes get prompted to choose a different encoding when I send
email responses to mail that looked pure ASCII.  I'm presented with a
buffer that points to a character that doesn't look "foreign" at all.
If I care to investigate, it turns out that character was an em-dash
or fancy quotes, that just looked "normal" due to the font.  Am I
confused?  No.  I just select the encoding, usually the one suggested
by Emacs as the default, and that's it.

IOW, I see no issue here.  But since people make a lot of fuss out of
it, feel free to invent all kinds of weird functions to save them from
themselves.

Re: bidi-display-reordering is now non-nil by default

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

> I sometimes get prompted to choose a different encoding when I send
> email responses to mail that looked pure ASCII.  I'm presented with a
> buffer that points to a character that doesn't look "foreign" at all.
> If I care to investigate, it turns out that character was an em-dash
> or fancy quotes, that just looked "normal" due to the font.  Am I
> confused?  No.  I just select the encoding, usually the one suggested
> by Emacs as the default, and that's it.
>
> IOW, I see no issue here.  But since people make a lot of fuss out of
> it, feel free to invent all kinds of weird functions to save them from
> themselves.

The issue is not saving them from themselves, but saving them and
anybody they might be sharing messages and data with from Emacs.

-- 
David Kastrup

Re: bidi-display-reordering is now non-nil by default

[Stefan Monnier]

>> > When you type "C-x C-s", Emacs clearly shows the characters that
>> > cannot be encoded as Latin-1.
>> As you said yourself, users should not need to know about the details of
>> how bidi works.  So we can't assume that they know what left-to-right
>> marks are, so they can't know whether the characters that Emacs is
>> complaining about are important to keep.
> I sometimes get prompted to choose a different encoding when I send
> email responses to mail that looked pure ASCII.  I'm presented with a
> buffer that points to a character that doesn't look "foreign" at all.
> If I care to investigate, it turns out that character was an em-dash
> or fancy quotes, that just looked "normal" due to the font.  Am I
> confused?  No.  I just select the encoding, usually the one suggested
> by Emacs as the default, and that's it.
> IOW, I see no issue here.  But since people make a lot of fuss out of
> it, feel free to invent all kinds of weird functions to save them from
> themselves.

And AFAIK that only affects Rmail users.  Other MUAs will just happily
choose another encoding (typically utf-8) for you.


        Stefan

Re: bidi-display-reordering is now non-nil by default

[Lars Magne Ingebrigtsen]

Chong Yidong <cyd@stupidchicken.com> writes:

> + 	    (if gnus-tmp-subject-or-nil
> + 		(setq gnus-tmp-subject-or-nil
> + 		      (string-mark-left-to-right gnus-tmp-subject-or-nil)))

This won't work (reliably), since the subject may be length-limited at a
later point.  But it'll work in the default format case, since the
subject there isn't length-limited.

The same thing applies to the From header -- which is length-limited by
default, so it's better to put the LRM marking further down into the
format-spec machinery.  But somehow just applying them to From/Subject,
since calling it for every item would be kinda slow...

It's still rather vague to me what the best way to handle this is, but
it should be doable...

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: bidi-display-reordering is now non-nil by default

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> It's still rather vague to me what the best way to handle this is, but
> it should be doable...

Ah.  I'll just amend `gnus-tilde-max-form' to check whether the last
character is LRM, and if it is, then it'll re-add it to the end of the
string if it decides to chop off the string.  I think that should work
reliably. 

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: bidi-display-reordering is now non-nil by default

[Lars Magne Ingebrigtsen]

[-- Attachment #1: Type: text/plain, Size: 503 bytes --]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Ah.  I'll just amend `gnus-tilde-max-form' to check whether the last
> character is LRM, and if it is, then it'll re-add it to the end of the
> string if it decides to chop off the string.  I think that should work
> reliably. 

I've now done this for the "from" part of the summary lines as a test.
As you can see from the attachment, my Hebrew-ish name gets chopped and
LRM-eyd correctly, as far as I can see, so it seems like a workable
approach:


[-- Attachment #2: a --]
[-- Type: application/octet-stream, Size: 57 bytes --]

!  [   5: Lars Magne שלום Ingebri‎] This is a test

[-- Attachment #3: Type: text/plain, Size: 468 bytes --]


However, this feels like an extremely leaky, er, abstraction.  I think
it would be better if all y'all can find a solution that doesn't require
these extreme low-level hacks just to get text rendered properly.  I
kinda doubt that all package maintainers will expend the hackery time
needed to twiddle these knobs.

But for Emacs 24, that's probably not realistic?

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Date: Thu, 18 Aug 2011 01:14:40 +0200
> 
> I've now done this for the "from" part of the summary lines as a test.
> As you can see from the attachment, my Hebrew-ish name gets chopped and
> LRM-eyd correctly, as far as I can see, so it seems like a workable
> approach:
> 
> !  [   5: Lars Magne שלום Ingebri‎] This is a test

It looks okay, but that's not the real test, because it will render
correctly even without the LRM.  The real test is to have your "from"
_end_ with strong R characters, and have the next field in the summary
begin with weak characters, like digits.  For example, remove the
"Ingebrigtsen" part, and start the Subject with "123".  Then try that
with and without the LRM.

> However, this feels like an extremely leaky, er, abstraction.  I think
> it would be better if all y'all can find a solution that doesn't require
> these extreme low-level hacks just to get text rendered properly.  I
> kinda doubt that all package maintainers will expend the hackery time
> needed to twiddle these knobs.

I'd appreciate ideas.  We currently have these:

  . Use LRM at the end of the field

  . Use TABs between fields

  . Replace each field with a display string whose value is the field
    text, with the disadvantage that cursor motion across the field
    and editing of the fields' text will need to be explicitly coded

  . Add a feature where a small set of characters can be requested by
    the Lisp application to behave like segment separator (TAB), then
    use those to separate fields, with the caveat that those separator
    characters should never appear as part of fields themselves

> But for Emacs 24, that's probably not realistic?

Depends on the idea, I guess.

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Date: Thu, 18 Aug 2011 00:25:20 +0200
> 
> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
> 
> > It's still rather vague to me what the best way to handle this is, but
> > it should be doable...
> 
> Ah.  I'll just amend `gnus-tilde-max-form' to check whether the last
> character is LRM, and if it is, then it'll re-add it to the end of the
> string if it decides to chop off the string.

Please do that also for RLM and PDF, not just LRM.  Any one of them,
if present at the end of the string, should be re-added to the end of
the chopped substring, because otherwise the result on display may be
adversely affected.

Re: bidi-display-reordering is now non-nil by default

[Lars Magne Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Please do that also for RLM and PDF, not just LRM.  Any one of them,
> if present at the end of the string, should be re-added to the end of
> the chopped substring, because otherwise the result on display may be
> adversely affected.

Ok; will do.  What are the RLM and PDF characters?  I mean, how do I
find them on the keyboard so I can insert them into the Gnus source
code...  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Sat, 10 Sep 2011 21:11:00 +0200
> 
> What are the RLM and PDF characters?  I mean, how do I find them on
> the keyboard so I can insert them into the Gnus source code...  :-)

C-x 8 RET right-to- TAB m TAB RET
C-x 8 RET pop TAB RET

Re: bidi-display-reordering is now non-nil by default

[Lars Magne Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> C-x 8 RET right-to- TAB m TAB RET
> C-x 8 RET pop TAB RET

Thanks; I've now included those chars, too.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: bidi-display-reordering is now non-nil by default

[Lars Magne Ingebrigtsen]

Chong Yidong <cyd@stupidchicken.com> writes:

> Still, it seems better not to change Gnus to proactively insert LRM
> characters, but leave it to those users who care to customize it as
> necessary.  In the default value of gnus-summary-line-format,
>
>   "%U%R%z%I%(%[%4L: %-23,23f%]%) %s\n"
>
> the subject (%s) is followed by a newline.  If a user wants to change
> this to include, say, the article number, that could be done via
>
>   "%U%R%z%I%(%[%4L: %-23,23f%]%) %s\u200E%N\n"

After reading through (ok, skimming :-) this thread, I'm also leaning
towards this as the solution (or "solution") for Emacs 24.1 at least.

However, there are complications.  :-)

Basically, people compose their own summary lines.  Even if we do what
Eli suggests, and default to putting in directional characters by
default, this won't really help the people who have customised their
summary buffers.

The additional additional complexity here is that the format specifiers
are complicated in and of themselves.

If we look at the typical elements in a summary line, it's pretty l2r,
except the "from" and "subject" elements, which may be l2r or r2l.  It's
not clear to me how this partitioning would be expressed reliably with
characters.

Take the default "from" spec, which is this: "%-23,23f".  This means
"pad to 23 characters, and chop off bits that are longer than 23
characters".  The "from" is available in a dynamic variable that's
inserted in the buffer by the format-spec machinery.  If I pad the
"from" with segmentation charaters at the start and end, so that
`gnus-tmp-from' becomes "\200Lars Lalala Ingebrigtsen\200", the last
directional character will be chopped off, and the bidi machinery will
wreak havoc with the characters that come afterwards.

In other words, the segmentation characters probably has to be inserted
in the format spec explicitly by users that customise this thing.  Which
just seems like a bad interface.

(And I haven't even mentioned the extra problems that comes with the
code that generates the summary lines is actually stored in each user's
.newsrc.eld file, so even if we somehow find a way to change the
format-spec machinery to be able to "mark" special specifiers to (sort
of) add segmentation characters "around" them, we're not totally out of
the woods.  But that could be solved somehow, I guess.)

So, how would I solve this conundrum?  I'd use text properties.  :-)

That is, Gnus would propertise `gnus-tmp-from' with `line-segment'
`from'.  The rendering engine could then look at line segments
(i.e. consecutive ranges of characters with identical `line-segment'
text props) and do the bidi thing on each segment separately. 

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: bidi-display-reordering is now non-nil by default

[David Kastrup]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Chong Yidong <cyd@stupidchicken.com> writes:
>
>> Still, it seems better not to change Gnus to proactively insert LRM
>> characters, but leave it to those users who care to customize it as
>> necessary.  In the default value of gnus-summary-line-format,
>>
>>   "%U%R%z%I%(%[%4L: %-23,23f%]%) %s\n"
>>
>> the subject (%s) is followed by a newline.  If a user wants to change
>> this to include, say, the article number, that could be done via
>>
>>   "%U%R%z%I%(%[%4L: %-23,23f%]%) %s\u200E%N\n"
>
> After reading through (ok, skimming :-) this thread, I'm also leaning
> towards this as the solution (or "solution") for Emacs 24.1 at least.
>
> However, there are complications.  :-)

> In other words, the segmentation characters probably has to be
> inserted in the format spec explicitly by users that customise this
> thing.  Which just seems like a bad interface.

Well, then how about letting format do the job?  It could make sure that
%s does not spill its direction into the surroundings, by enclosing it
with appropriate marks if required.

Basically format first analyses the reading direction of each character
of the format string, and when a format specification would change this
direction after filling it, it gets surrounded by appropriate
directional marks that keep it from spilling.

If this seems too automagic, one could use format specs like %| to
indicate points where format could insert directional specifiers that
keep directionality from moving across.

-- 
David Kastrup

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  list-general@mohsen.1.banan.byname.net,  emacs-devel@gnu.org
> Date: Sat, 06 Aug 2011 17:51:51 +0200
> 
> Basically, people compose their own summary lines.  Even if we do what
> Eli suggests, and default to putting in directional characters by
> default, this won't really help the people who have customised their
> summary buffers.

Sorry, I don't understand.  Please tell the details.  You are talking
to a man who doesn't use Gnus.  What kinds of customizations are you
talking about?

> Take the default "from" spec, which is this: "%-23,23f".  This means
> "pad to 23 characters, and chop off bits that are longer than 23
> characters".  The "from" is available in a dynamic variable that's
> inserted in the buffer by the format-spec machinery.  If I pad the
> "from" with segmentation charaters at the start and end, so that
> `gnus-tmp-from' becomes "\200Lars Lalala Ingebrigtsen\200", the last
> directional character will be chopped off, and the bidi machinery will
> wreak havoc with the characters that come afterwards.

We need to add the directional characters after formatting.  Or modify
the ``format-spec machinery'', whatever that is, to remove characters
in-between, but not the directional marks themselves, or to insert the
marks as part of the formatting in the first place.  If none of that
can fix the problem, please tell the details of why not.

> So, how would I solve this conundrum?  I'd use text properties.  :-)
> 
> That is, Gnus would propertise `gnus-tmp-from' with `line-segment'
> `from'.  The rendering engine could then look at line segments
> (i.e. consecutive ranges of characters with identical `line-segment'
> text props) and do the bidi thing on each segment separately. 

This is a non-starter: as I explained elsewhere in this thread, the
reordering code completely ignores text properties.  It just finds the
next character to display; the properties are thereafter considered by
the code that calls the reordering engine.

Re: bidi-display-reordering is now non-nil by default

[Mohsen BANAN]

>>>>> On Fri, 05 Aug 2011 17:54:54 -0400, Chong Yidong <cyd@stupidchicken.com> said:

  Chong> Still, it seems better not to change Gnus to proactively insert LRM
  Chong> characters, but leave it to those users who care to customize it as
  Chong> necessary.  In the default value of gnus-summary-line-format,

  Chong>   "%U%R%z%I%(%[%4L: %-23,23f%]%) %s\n"

  Chong> the subject (%s) is followed by a newline.  If a user wants to change
  Chong> this to include, say, the article number, that could be done via

  Chong>   "%U%R%z%I%(%[%4L: %-23,23f%]%) %s\u200E%N\n"

  Chong> This has the advantage of avoiding inserting the insertion of excess LRM
  Chong> characters by default.  The disadvantage, of course, is that it requires
  Chong> some extra knowledge from users, but we can handle this by adding a note
  Chong> to the docstring.

I am opposed to the direction that you are
advocating.

Instead of discussing merits of tactics, I want us
to first step back and look at where we are and
what we are trying to do.

The topic at hand is:

    Making emacs24 packages bidi-aware

In a previous message a while back, I attempted to
move things towards formulation/articulation of
goals, policies and strategies. My hope was that
those playing a leadership role would build on
that.

Permit me to attempt in formulating some policies
for making emacs24 packages bidi-aware.

 A- We make a distinction between
   receiving/displaying bidi-awareness and 
   sending/originating bidi-awareness.

 B- We consider displaying bidi-awareness of
   emacs24 packages more urgent and more important.

 C- Anytime that an emacs24 package receives
   bidi-input that results in display problems, 
   we consider that a bug and file a bug report.

 D- With respect to displaying, at package level we
   attempt to eliminate bidi specific options.
   Displaying bidi is viewed as omni-present
   through emacs. It is not a feature to be
   opted-in or opted-out -- or require special
   configuration.
 
 E- With respect to originating bidi-awareness,
   bidi specific options are perfectly reasonable.

Now, assuming that the above policies are
reasonable, here are the problems that I see with
the direction that you advocated:

  Chong> Still, it seems better not to change Gnus to proactively insert LRM
  Chong> characters, but leave it to those users who care to customize it as
  Chong> necessary.  In the default value of gnus-summary-line-format,

So, 

  1) You are saying users "who care" must do
     bidi-specific customization to get
     bidi-display to work in Gnus. Which conflicts
     with my (D).

  2) You are saying that it is acceptable to not
     have the LRM for the subject line. In which
     case when a bidi message arrives display may
     be messed up (C).

     Jason pointed to this previously:

  Jason> Users who get a lot of RTL email may be motivated enough to do that, but
  Jason> users who seldom get such mail (and what they do get is probably mostly
  Jason> spam) will still see the muddled up
  Jason> summary line as a bug.

     It is not reasonable for us to accept emacs24
     to only be working right some of the time.

What you suggested, is a neat temporary hack. I
already changed mine to:

(setq gnus-summary-line-format ":%U%R %B %s %-60=\u200E|%4L |%-20,20f\u200E|%&user-date; \n")

And things are lining up better. Thanks.

But let's not just aim for temporary hacks.  Let's
try to eliminate all necessary complexities --
particularly when they require user exposure.

Permit me to also suggest some strategies for
making emacs24 packages bidi-aware.

  - We first put on the table several package use
    cases to bring out problem patterns.

    Thus far, as usage cases we have:
    
      - Gnus Summary mode

      - mail citations

      - XML mode --  Michael Duggan'si nfo was great

    I'd like to suggest a couple more. Bidi
    ramifications to auctex mode. And perhaps
    org's todo. I hope to be following up with
    these soon.

  - These usage patterns will likely demonstrate a
    need for something like a bidi-kit.el Where
    say insertion of LRM/... at string or point or
    ... gets common code. And also become a
    facility for bidi-menu.el features that were
    previously discussed.

  - Based on something like bidi-kit.el provide
    some examples so that in situations where the
    package maintainer is not motivated someone
    else can take care of moving the package
    forward towards more bidi-awareness.

    In other words we need to spread the knowledge
    for making emacs24 packages bidi-aware.

Emacs24 is now in feature freeze stage.  What we
have is what we have. And with what we have, many
bidi-display problems can be properly fixed. It is
a matter of doing it.

In summary, I am opposed to considering proper
display of bidi by packages as an option requiring
special configuration.

...Mohsen

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Sun, 31 Jul 2011 15:25:37 +0200
> 
> If we don't have the infrastructure for supporting a default setting in
> a way tolerable for most users, the default needs to be different until
> we do.

If you want to argue that the default setting is "intolerable for most
users", you will have to come up with more evidence than just 2 very
similar use cases (i.e. summary buffer of a MUA).  Emacs supports much
more than just summaries of email messages.

It would also be prudent to wait a bit more than a few hours, before
such far-reaching conclusions are made.  Let's collect a few more data
points, at the very least.

Re: bidi-display-reordering is now non-nil by default

[Stefan Monnier]

I think it makes sense to change the default value of
bidi-paragraph-direction to left-to-right.
I expect there are fewer modes where bidi-paragraph-direction needs to
be set to nil than modes where it needs to be set to left-to-right.
Furthermore a left-to-right default value is safer in the sense of
having better backward compatibility and avoiding pathological
performance problems.


        Stefan

Re: bidi-display-reordering is now non-nil by default

[Lars Magne Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> I think it makes sense to change the default value of
> bidi-paragraph-direction to left-to-right.

Would this mean that a (say) Gnus summary buffer with right-to-left
subjects would be rendered incorrectly?

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: bidi-display-reordering is now non-nil by default

[David Kastrup]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>> I think it makes sense to change the default value of
>> bidi-paragraph-direction to left-to-right.
>
> Would this mean that a (say) Gnus summary buffer with right-to-left
> subjects would be rendered incorrectly?

No.

With bidi-paragraph-direction left-to-right, material is arranged such
that the occupied area has the left screen border as one limit.  Set
right-to-left, the lines are oriented at the right screen border.  Set
to nil, the first directional character in the paragraph decides where
the paragraph is headed.

The subjects are not the first directional characters in the paragraph
of a summary buffer which consists of lines like

R.+[  12: Stefan Monnier         ] Re: bidi-display-reordering is now non-nil by default

so there is quite little incentive to make bidi-paragraph-direction
nil.  Any R2L material will still appear R2L, but it will cluster itself
to the left of the available area rather than the right.

A setting of anything but left-to-right would only make sense in the
Summary buffer after Gnus was heavily localized, using R2L or
nondirectional characters for its flag display, line counts and so on,
and consequently starting in a somewhat natural way at the right screen
border.

-- 
David Kastrup

Re: bidi-display-reordering is now non-nil by default

[Lars Magne Ingebrigtsen]

David Kastrup <dak@gnu.org> writes:

> The subjects are not the first directional characters in the paragraph
> of a summary buffer which consists of lines like
>
> R.+[ 12: Stefan Monnier ] Re: bidi-display-reordering is now non-nil
> by default
>
> so there is quite little incentive to make bidi-paragraph-direction
> nil.  Any R2L material will still appear R2L, but it will cluster itself
> to the left of the available area rather than the right.

I see.  Thanks for the explanation.

I guess that in Gnus (for instance), the only buffer it would make sense
to have `bidi-paragraph-direction' nil would be in the article buffer.
All the other buffers are (by default) quite left-to-right (displaying
summary lines, group lines, etc).

I think.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: bidi-display-reordering is now non-nil by default

[James Cloos]

>>>>> "DK" == David Kastrup <dak@gnu.org> writes:

DK> The subjects are not the first directional characters in the
DK> paragraph of a summary buffer which consists of lines like

DK> R.+[  12: Stefan Monnier         ] Re: bidi-display-reordering is now non-nil by default

The problem for gnus' summary buffer is the name of the poster of the
first message in said buffer.  A fresh summary buffer usually has only
neutral characters before the name, and r2l names do affect the entire
buffer until and unless one reads or deletes that first message, thereby
replacine a space with, typically, one of [dDER].

I haven't yet tried setting bidi-paragraph-direction in such a buffer.
(None have come up for me since I leaned of that var.)  That might fix
things for Summary.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
> Date: Tue, 02 Aug 2011 18:09:05 +0200
> 
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
> 
> > I think it makes sense to change the default value of
> > bidi-paragraph-direction to left-to-right.
> 
> Would this mean that a (say) Gnus summary buffer with right-to-left
> subjects would be rendered incorrectly?

Not "incorrectly", in any case.  In the summary buffer, it is actually
precisely what is needed.  But in other Gnus buffers, notably the ones
that show the text of messages and articles, the display will be less
than optimal, when the message is mostly composed of R2L text.  Still
legible, though.  These Gnus buffers will need to reset the paragraph
direction to nil, to present the text in an optimal manner.

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Tue, 02 Aug 2011 11:34:21 -0400
> 
> I think it makes sense to change the default value of
> bidi-paragraph-direction to left-to-right.
> I expect there are fewer modes where bidi-paragraph-direction needs to
> be set to nil than modes where it needs to be set to left-to-right.
> Furthermore a left-to-right default value is safer in the sense of
> having better backward compatibility and avoiding pathological
> performance problems.

I don't necessarily disagree, and doing that is a one-line change.
But let's not do it just yet, for two reasons.

First, we only discovered 2 kinds of affected modes for now: Org mode
and the mail/news summary modes.  So our expectations of the number of
affected modes do not have enough data points to back them up.  I
think we should gather more, to be able to weigh the merits and
demerits of this.  Eventually, some modes will need to use non-default
values of this variable, so it really is an issue with how many modes
will have to do this or that change.

Second, concealing the potential performance problems at this early
stage would make the testing of the bidi code much less effective,
because evidently it was not used enough in the 1.5 years it was
available on the trunk, if such problems are only now get reported.
The problem with Org mode, for example, made it obvious that one of
the optimizations I introduced to prevent slowdowns is failing in
several important use cases, and needs to be redesigned.  If the
paragraph direction in Org buffers was forced to left-to-right, who
knows when this grave problem would be discovered and reported.

The "backward compatibility problems" part, I don't understand.  Can
you explain what you are alluding to?

Re: bidi-display-reordering is now non-nil by default

[Chong Yidong]

Eli Zaretskii <eliz@gnu.org> writes:

> Chong, Stefan: how about having a (customizable) list of buffer names
> which should by default have left-to-right paragraph direction?  Would
> this be a reasonable solution?  We could add buffer names to this list
> as we discover more such cases.

We could introduce a customizable variable, but I think this variable
should serve only as a fallback for individual users---it should be nil
by default.  Otherwise that discourages upstream authors from
implementing a real fix.

I don't know enough to say what the real fix is, though.  You said that
the slowdown is because "Emacs is searching for the beginning of the
paragraph, but this whole buffer is one huge paragraph."  Is it enough
for a mode to set the variable `paragraph-start' for Emacs to DTRT?  A
quick glance at the Mew source shows that it does set `paragraph-start',
though I don't know if every line is treated like a paragraph.

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Chong Yidong <cyd@stupidchicken.com>
> Cc: Werner LEMBERG <wl@gnu.org>, emacs-devel@gnu.org
> Date: Mon, 01 Aug 2011 11:45:54 -0400
> 
> I don't know enough to say what the real fix is, though.  You said that
> the slowdown is because "Emacs is searching for the beginning of the
> paragraph, but this whole buffer is one huge paragraph."  Is it enough
> for a mode to set the variable `paragraph-start' for Emacs to DTRT?

No.  The bidi reordering doesn't use the value of `paragraph-start',
because some modes (e.g., Texinfo) set it to something utterly
incompatible with what "paragraph" means in the UAX#9 context.
(Originally, the code did use `paragraph-start', but that caused bad
crashes, so I backed up.)

In general, a "bidi paragraph" must begin after a newline, or else the
Unicode Bidirectional Algorithm, which Emacs implements, would need
serious changes, with the result that compatibility between Emacs and
other bidi-supporting applications would be questionable at best.

We could introduce a separate variable for a "bidi paragraph" start
and separator, but I never tried to see whether the current code will
work well with an arbitrary value of such a variable, not even if its
value is a single newline (i.e. each line is a separate paragraph).
I'd rather not start such an adventure now.

> A quick glance at the Mew source shows that it does set
> `paragraph-start', though I don't know if every line is treated like
> a paragraph.

The issue at hand is with the Mew Summary buffer, not with the buffer
where it displays the mail messages themselves.

Re: bidi-display-reordering is now non-nil by default

[Lars Ingebrigtsen]

Mind you, I don't know anything about bidi, but I'm just throwing this
out there.  Feel free to ignore.

Would it possibly, perhaps, make sense to switch off the bidi paragraph
detection stuff if (get major-mode 'mode-class) eq 'special?  These
modes aren't meant for editing (generally), so perhaps that might solve
some problems.

Or not.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Mon, 01 Aug 2011 22:33:49 +0200
> 
> Would it possibly, perhaps, make sense to switch off the bidi paragraph
> detection stuff if (get major-mode 'mode-class) eq 'special?

What do you mean by "switch off"? do you mean force paragraphs to be
always left-to-right?

> These modes aren't meant for editing (generally), so perhaps that
> might solve some problems.

It should be possible, but:

 . To say something intelligent about this, I'd need someone "in the
   know" to explain what `special-mode' really means.  I actually
   thought about this a day or two ago, but was unable to make up my
   mind due to lack of detailed documentation.  The ELisp manual says
   this about these modes:

    * If this mode is appropriate only for specially-prepared text, then
      the major mode command symbol should have a property named
      `mode-class' with value `special', put on as follows:

	   (put 'funny-mode 'mode-class 'special)

      This tells Emacs that new buffers created while the current buffer
      is in Funny mode should not inherit Funny mode, in case the
      default value of `major-mode' is `nil'.  Modes such as Dired,
      Rmail, and Buffer List use this feature.

   Huh?  "New buffers created while the current buffer is in Funny
   mode"?  "Should not inherit Funny mode"?  "In case the default
   value is non-nil"?  I cannot make heads or tails out of this
   description, and saying that a few modes use this feature really
   doesn't help to understand it.

 . One of the suggestions on the table is that
   bidi-paragraph-direction be set to left-to-right by default, and
   then only major modes (or other features) that need to reset it to
   nil will do that.  If the decision is to go this way, we won't need
   to treat `special' modes "specially" ;-)

Btw, did anyone fix Gnus summary, to display the line count correctly
when the Subject is using an R2L script?  It was reported a day or two
ago here, and I said that the LRM character (possibly invisible)
should be inserted right after the subject to fix that.

Re: bidi-display-reordering is now non-nil by default

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Lars Ingebrigtsen <larsi@gnus.org>
>> Date: Mon, 01 Aug 2011 22:33:49 +0200
>> 
>> Would it possibly, perhaps, make sense to switch off the bidi paragraph
>> detection stuff if (get major-mode 'mode-class) eq 'special?
>
> What do you mean by "switch off"? do you mean force paragraphs to be
> always left-to-right?
>
>> These modes aren't meant for editing (generally), so perhaps that
>> might solve some problems.
>
> It should be possible, but:
>
>  . To say something intelligent about this, I'd need someone "in the
>    know" to explain what `special-mode' really means.  I actually
>    thought about this a day or two ago, but was unable to make up my
>    mind due to lack of detailed documentation.  The ELisp manual says
>    this about these modes:
>
>     * If this mode is appropriate only for specially-prepared text, then
>       the major mode command symbol should have a property named
>       `mode-class' with value `special', put on as follows:
>
> 	   (put 'funny-mode 'mode-class 'special)
>
>       This tells Emacs that new buffers created while the current buffer
>       is in Funny mode should not inherit Funny mode, in case the
>       default value of `major-mode' is `nil'.  Modes such as Dired,
>       Rmail, and Buffer List use this feature.
>
>    Huh?  "New buffers created while the current buffer is in Funny
>    mode"?  "Should not inherit Funny mode"?  "In case the default
>    value is non-nil"?  I cannot make heads or tails out of this
>    description, and saying that a few modes use this feature really
>    doesn't help to understand it.

"Funny mode" is a place holder, something like $mode.

Anyway, the key point is "specially-prepared", meaning that the skeleton
of the text is provided by the mode itself.  Given that the standard
user interface of Emacs is English, left-to-right mode as a default
seems reasonable.  Should the specially prepared text be R2L (namely a
user interface _not_ in English, or customizable), the mode writer would
be aware of it.

-- 
David Kastrup

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Tue, 02 Aug 2011 09:20:24 +0200
> 
> >     * If this mode is appropriate only for specially-prepared text, then
> >       the major mode command symbol should have a property named
> >       `mode-class' with value `special', put on as follows:
> >
> > 	   (put 'funny-mode 'mode-class 'special)
> >
> >       This tells Emacs that new buffers created while the current buffer
> >       is in Funny mode should not inherit Funny mode, in case the
> >       default value of `major-mode' is `nil'.  Modes such as Dired,
> >       Rmail, and Buffer List use this feature.
> >
> >    Huh?  "New buffers created while the current buffer is in Funny
> >    mode"?  "Should not inherit Funny mode"?  "In case the default
> >    value is non-nil"?  I cannot make heads or tails out of this
> >    description, and saying that a few modes use this feature really
> >    doesn't help to understand it.
> 
> "Funny mode" is a place holder, something like $mode.

Yes, I understood that, but the text still made no sense.

> Anyway, the key point is "specially-prepared", meaning that the skeleton
> of the text is provided by the mode itself.

Thanks, I begin to see the light now.  But still, what does the manual
mean by "should not inherit", and what is the importance of the
default value of major-mode?

> Given that the standard user interface of Emacs is English,
> left-to-right mode as a default seems reasonable.  Should the
> specially prepared text be R2L (namely a user interface _not_ in
> English, or customizable), the mode writer would be aware of it.

Yes, I agree.

So if we don't decide to set the paragraph direction L2R by default, I
will look into setting it in any `special' mode (among others).

Thanks.

Re: bidi-display-reordering is now non-nil by default

[Štěpán Němec]

On Tue, 02 Aug 2011 09:43:32 +0200
Eli Zaretskii wrote:

>> From: David Kastrup <dak@gnu.org>
>> Date: Tue, 02 Aug 2011 09:20:24 +0200
>> 
>> >     * If this mode is appropriate only for specially-prepared text, then
>> >       the major mode command symbol should have a property named
>> >       `mode-class' with value `special', put on as follows:
>> >
>> > 	   (put 'funny-mode 'mode-class 'special)
>> >
>> >       This tells Emacs that new buffers created while the current buffer
>> >       is in Funny mode should not inherit Funny mode, in case the
>> >       default value of `major-mode' is `nil'.  Modes such as Dired,
>> >       Rmail, and Buffer List use this feature.
>> >
>> >    Huh?  "New buffers created while the current buffer is in Funny
>> >    mode"?  "Should not inherit Funny mode"?  "In case the default
>> >    value is non-nil"?  I cannot make heads or tails out of this
>> >    description, and saying that a few modes use this feature really
>> >    doesn't help to understand it.
>> 
>> "Funny mode" is a place holder, something like $mode.
>
> Yes, I understood that, but the text still made no sense.
>
>> Anyway, the key point is "specially-prepared", meaning that the skeleton
>> of the text is provided by the mode itself.
>
> Thanks, I begin to see the light now.  But still, what does the manual
> mean by "should not inherit", and what is the importance of the
> default value of major-mode?

What about having a look at the relevant documentation? ;-)

(describe-variable 'major-mode):

Symbol for current buffer's major mode.
The default value (normally `fundamental-mode') affects new buffers.
A value of nil means to use the current buffer's major mode, provided
it is not marked as "special".

-- 
Štěpán

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Štěpán Němec <stepnem@gmail.com>
> Cc: David Kastrup <dak@gnu.org>,  emacs-devel@gnu.org
> Date: Tue, 02 Aug 2011 12:27:18 +0200
> 
> > But still, what does the manual mean by "should not inherit", and
> > what is the importance of the default value of major-mode?
> 
> What about having a look at the relevant documentation? ;-)
> 
> (describe-variable 'major-mode):
> 
> Symbol for current buffer's major mode.
> The default value (normally `fundamental-mode') affects new buffers.
> A value of nil means to use the current buffer's major mode, provided
> it is not marked as "special".

Who could have thought it is relevant?

Thanks.

Re: bidi-display-reordering is now non-nil by default

[Lars Magne Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Btw, did anyone fix Gnus summary, to display the line count correctly
> when the Subject is using an R2L script?  It was reported a day or two
> ago here, and I said that the LRM character (possibly invisible)
> should be inserted right after the subject to fix that.

If people cut and paste summary lines, won't the LRM character then be
included in what they cut and paste?  It's just my gut reaction, but
inserting invisible characters into buffers seems like a non-optimal
design to me...

Could possibly the LRM control instead be done with text properties or
the like?

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Date: Tue, 02 Aug 2011 15:37:41 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Btw, did anyone fix Gnus summary, to display the line count correctly
> > when the Subject is using an R2L script?  It was reported a day or two
> > ago here, and I said that the LRM character (possibly invisible)
> > should be inserted right after the subject to fix that.
> 
> If people cut and paste summary lines, won't the LRM character then be
> included in what they cut and paste?

Not by default: invisible is in yank-excluded-properties, so the LRM
character will become visible after yanking.

> It's just my gut reaction, but inserting invisible characters into
> buffers seems like a non-optimal design to me...

Then don't make it invisible, it displays by default as a very thin
space character.  You can see that in action in the HELLO file, on the
Arabic and Hebrew lines.

> Could possibly the LRM control instead be done with text properties or
> the like?

No, and doing that is a very bad idea.  For starters, it means that if
you send such a copy/pasted text to someone else through email, they
will see the text differently than you did.  Using LRM is the only
idea that works well and without complications with plain text.

Re: bidi-display-reordering is now non-nil by default

[Lars Magne Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> It's just my gut reaction, but inserting invisible characters into
>> buffers seems like a non-optimal design to me...
>
> Then don't make it invisible, it displays by default as a very thin
> space character.  You can see that in action in the HELLO file, on the
> Arabic and Hebrew lines.
>
>> Could possibly the LRM control instead be done with text properties or
>> the like?
>
> No, and doing that is a very bad idea.  For starters, it means that if
> you send such a copy/pasted text to someone else through email, they
> will see the text differently than you did.  Using LRM is the only
> idea that works well and without complications with plain text.

I may have misinterpreted you, but I thought you meant that Gnus should
put LRM character(s) into the summary buffer (by default) to signal that
the buffer really is quite left-to-right-ey.  I think that would be a
less than optimal way of specifying something like that.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Tue, 02 Aug 2011 19:35:12 +0200
> 
> I may have misinterpreted you, but I thought you meant that Gnus should
> put LRM character(s) into the summary buffer (by default) to signal that
> the buffer really is quite left-to-right-ey.  I think that would be a
> less than optimal way of specifying something like that.

No, it's my fault: I didn't really explain the issue.  It's not with
the buffer being L2R.

Here's the part of a Gnus summary buffer cited by Mohsen:

---- Gnus Summary Sample Begin ----
:!   2 moons in the sky , August the 27th, يک آسمان و دو ماه|  37 |First LAST           |Jul 13 '10 
:!   Fw: بسیار جالب و خواندنی است ... فقط با دقت و حوصله بخو| 284 |First LAST           |14:15 
:!   Re: Next Saturday instead of this weekend              |  36 |First LAST           |14:22 
---- Gnus Summary Sample Ends  ----

Note that the line counts of the first two summary lines are
misplaced: they appear _to_the_left_ of the subject (which is the part
written in Farsi).  The problem here is that a number (a string of
digits) that follows R2L text gets displayed to the left of that text,
according to the UBA (the Unicode Bidirectional Algorithm), on the
assumption that it is part of the R2L text.  The `|' separator is a
weak character, so it doesn't change that order.

To fix the alignment, you need to append the LRM to each subject:

:!   2 moons in the sky , August the 27th, يک آسمان و دو ماه‎|  37 |First LAST           |Jul 13 '10 
:!   Fw: بسیار جالب و خواندنی است ... فقط با دقت و حوصله بخو‎| 284 |First LAST           |14:15 
:!   Re: Next Saturday instead of this weekend             ‎|  36 |First LAST           |14:22 

See how the line counts are nicely aligned now?  I inserted an LRM
just before the `|' separator to the left of the line count.  I did
that in the last line as well, but there it doesn't matter because the
subject uses only L2R text.

To summarize, Gnus should append the LRM to each and every subject it
puts in the summary buffer.

Did I explain the issue now?

Re: bidi-display-reordering is now non-nil by default

[محسن بنان]

>>>>> On Tue, 02 Aug 2011 22:44:16 +0300, Eli Zaretskii <eliz@gnu.org> said:

  Eli> To fix the alignment, you need to append the LRM to each subject:

  Eli> :!   2 moons in the sky , August the 27th, يک آسمان و دو ماه‎|  37 |First LAST           |Jul 13 '10 
  Eli> :!   Fw: بسیار جالب و خواندنی است ... فقط با دقت و حوصله بخو‎| 284 |First LAST           |14:15 
  Eli> :!   Re: Next Saturday instead of this weekend             ‎|  36 |First LAST           |14:22 

  Eli> See how the line counts are nicely aligned now?  I inserted an LRM
  Eli> just before the `|' separator to the left of the line count.  I did
  Eli> that in the last line as well, but there it doesn't matter because the
  Eli> subject uses only L2R text.

  Eli> To summarize, Gnus should append the LRM to each and every subject it
  Eli> puts in the summary buffer.

The same thing applies to the From Name.
First LAST in the above example.

I have writen my name in Farsi in the From: line
(and Cc: line) as an example.

Thanks.

...Mohsen

Re: bidi-display-reordering is now non-nil by default

[James Cloos]

> The same thing applies to the From Name.
> First LAST in the above example.

As I wrote earlier, gnus' summary buffer will become r2l with its
default summary line format whenever the first article's sender's
name is r2l and there are no marks, since everything before the
sender's name is neutral.

I did encounter an example shortly after I posted that.

Setting bidi-paragraph-direction to left-to-right for the Summary
buffer was indeed enough to render the buffer correctly.

I don't know whether any gnus users customize their Group and Summary
buffers to a non-English locale and would prefer a r2l buffer, but at
least for those of us in en locales setting bidi-paragraph-direction
to left-to-right in gnus' Groups and Summary buffers is the Right
Thing To Do.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: bidi-display-reordering is now non-nil by default

[Mohsen BANAN]

James, 

I think we are talking about two different things.

The topic here is not paragraph direction. 
The topic is alignment.

Eli described it in full, the summary is:

 Eli> To fix the alignment, you need to append the LRM to each subject:

Separate from that, the gnus-summary-line-format
should remain customizable. On a larger screen, mine is:

 (setq gnus-summary-line-format ":%U%R %B %s %-60=|%4L |%-20,20f |%&user-date; \n")

That means, directionality can be triggered by
different fields. 

Until it is possible to fully localize Gnus/Emacs,
bidi-paragraph-direction for Summary should *always*
be left-to-right. No questions there. 

The topic is alignment. The way to do it is LRM
as Eli described.

...Mohsen

>>>>> On Wed, 03 Aug 2011 01:11:56 -0400, James Cloos <cloos@jhcloos.com> said:

  >> The same thing applies to the From Name.
  >> First LAST in the above example.

  James> As I wrote earlier, gnus' summary buffer will become r2l with its
  James> default summary line format whenever the first article's sender's
  James> name is r2l and there are no marks, since everything before the
  James> sender's name is neutral.

  James> I did encounter an example shortly after I posted that.

  James> Setting bidi-paragraph-direction to left-to-right for the Summary
  James> buffer was indeed enough to render the buffer correctly.

  James> I don't know whether any gnus users customize their Group and Summary
  James> buffers to a non-English locale and would prefer a r2l buffer, but at
  James> least for those of us in en locales setting bidi-paragraph-direction
  James> to left-to-right in gnus' Groups and Summary buffers is the Right
  James> Thing To Do.

  James> -JimC
  James> -- 
  James> James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: James Cloos <cloos@jhcloos.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Lars Magne Ingebrigtsen <larsi@gnus.org>,  emacs-devel@gnu.org
> Date: Wed, 03 Aug 2011 01:11:56 -0400
> 
> > The same thing applies to the From Name.
> > First LAST in the above example.
> 
> As I wrote earlier, gnus' summary buffer will become r2l with its
> default summary line format whenever the first article's sender's
> name is r2l and there are no marks, since everything before the
> sender's name is neutral.
> 
> I did encounter an example shortly after I posted that.
> 
> Setting bidi-paragraph-direction to left-to-right for the Summary
> buffer was indeed enough to render the buffer correctly.

As Mohsen points out, this is not the issue at hand.  The issue at
hand is rendering individual summary lines that include portions, like
Subject and names of addressees, in R2L scripts.

> I don't know whether any gnus users customize their Group and Summary
> buffers to a non-English locale and would prefer a r2l buffer

IMO, such customizations don't make sense until the Emacs UI,
including the menu bar, the tool bar, and the mode line, can be
customized to display right to left.  There's a TODO item about that,
but it probably won't happen for Emacs 24.1.

Re: bidi-display-reordering is now non-nil by default

[James Cloos]

>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:

>> Setting bidi-paragraph-direction to left-to-right for the Summary
>> buffer was indeed enough to render the buffer correctly.

EZ> As Mohsen points out, this is not the issue at hand.  The issue at
EZ> hand is rendering individual summary lines that include portions,
EZ> like Subject and names of addressees, in R2L scripts.

If you write about something other than what I saw, it must only occur
with customized summary formats.  With the default format the *only*
problem was the overall buffer direction, and only when the first name
was r2l and there were no marks on that line.

Everything else displays here without error.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: James Cloos <cloos@jhcloos.com>
> Cc: list-general@mohsen.1.banan.byname.net,  larsi@gnus.org,  emacs-devel@gnu.org
> Date: Wed, 03 Aug 2011 08:08:30 -0400
> 
> If you write about something other than what I saw, it must only occur
> with customized summary formats.  With the default format the *only*
> problem was the overall buffer direction, and only when the first name
> was r2l and there were no marks on that line.
> 
> Everything else displays here without error.

Did you have email messages or articles whose Subject included R2L
letters?  The problem I was talking about happens only when the
Subject ends with R2L characters, and then the line count that follows
it in the summary display is aligned incorrectly.  Look at the example
posted by Mohsen that I quoted a few messages back.

Re: bidi-display-reordering is now non-nil by default

[James Cloos]

>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:

EZ> Did you have email messages or articles whose Subject included R2L
EZ> letters?

I just created one to test.  The Summary buffer looks correct.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: James Cloos <cloos@jhcloos.com>
> Cc: list-general@mohsen.1.banan.byname.net,  larsi@gnus.org,  emacs-devel@gnu.org
> Date: Wed, 03 Aug 2011 09:06:55 -0400
> 
> >>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:
> 
> EZ> Did you have email messages or articles whose Subject included R2L
> EZ> letters?
> 
> I just created one to test.  The Summary buffer looks correct.

Can you show me the relevant portion of the Summary buffer?

Re: bidi-display-reordering is now non-nil by default

[James Cloos]

>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:

EZ> Can you show me the relevant portion of the Summary buffer?

 . [   3: ➤ cloos@jhcloos.com    ] العربيه
 . [  14: Eli Zaretskii          ] Re: bidi-display-reordering is now non-nil by default
 . [ 399: Eli Zaretskii          ] Re: Memory leak due to bidi?

That was before reading the messages.

(As you can see from my test message, any messages From me show who they
are To; the rest show who they are From.)

Ordering by subject or date, ascending or descending did not affect the
display (beyond re-ordering the lines, of course).

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: James Cloos <cloos@jhcloos.com>
> Cc: list-general@mohsen.1.banan.byname.net,  larsi@gnus.org,  emacs-devel@gnu.org
> Copyright: Copyright 2011 James Cloos
> OpenPGP-Fingerprint: E9E9 F828 61A4 6EA9 0F2B  63E7 997A 9F17 ED7D AEA6
> Date: Wed, 03 Aug 2011 10:10:23 -0400
> 
> >>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:
> 
> EZ> Can you show me the relevant portion of the Summary buffer?
> 
>  . [   3: ➤ cloos@jhcloos.com    ] العربيه
>  . [  14: Eli Zaretskii          ] Re: bidi-display-reordering is now non-nil by default
>  . [ 399: Eli Zaretskii          ] Re: Memory leak due to bidi?
> 

Thanks.  The problem happens if the Subject is followed by a line
count of the message.  Perhaps that happens only if the summary format
is customized, I don't know Gnus enough to tell.

Re: bidi-display-reordering is now non-nil by default

[Mohsen BANAN]

>>>>> On Wed, 03 Aug 2011 19:08:39 +0300, Eli Zaretskii <eliz@gnu.org> said:

  >> From: James Cloos <cloos@jhcloos.com>
  >> Cc: list-general@mohsen.1.banan.byname.net,  larsi@gnus.org,  emacs-devel@gnu.org
  >> Copyright: Copyright 2011 James Cloos

What the heck is this -- Copyright? 

This list is in the realm of Copyleft.

  >> OpenPGP-Fingerprint: E9E9 F828 61A4 6EA9 0F2B  63E7 997A 9F17 ED7D AEA6
  >> Date: Wed, 03 Aug 2011 10:10:23 -0400
  >> 
  >> >>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:
  >> 
  EZ> Can you show me the relevant portion of the Summary buffer?
  >> 
  >> . [   3: ➤ cloos@jhcloos.com    ] العربيه
  >> . [  14: Eli Zaretskii          ] Re: bidi-display-reordering is now non-nil by default
  >> . [ 399: Eli Zaretskii          ] Re: Memory leak due to bidi?
  >> 

  Eli> Thanks.  The problem happens if the Subject is followed by a line
  Eli> count of the message.  Perhaps that happens only if the summary format
  Eli> is customized, I don't know Gnus enough to tell.

Yes. 

This is what is happening.

And in a previous message to James, I pointed
directly to this:

>>>>> On Tue, 02 Aug 2011 22:52:04 -0700, Mohsen BANAN <list-general@mohsen.1.banan.byname.net> said:

  Mohsen> James, 

  Mohsen> ...
   
  Mohsen> Separate from that, the gnus-summary-line-format
  Mohsen> should remain customizable. On a larger screen, mine is:

  Mohsen>  (setq gnus-summary-line-format ":%U%R %B %s %-60=|%4L |%-20,20f |%&user-date; \n")

  Mohsen> That means, directionality can be triggered by
  Mohsen> different fields. 

Therefore, for the Summary buffer paragraph
direction should be locked into left-to-right --
not nil.

And to get the formatting (alignment) right, any field
in gnus-summary-line-format that could be R2L
needs the LRM. As it can mess other fields -- as
previously shown.

Thanks.

...Mohsen

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: محسن بنان
>  <list-general@mohsen.1.banan.byname.net>
> Cc: Lars Magne Ingebrigtsen <larsi@gnus.org>,  emacs-devel@gnu.org,
>   محسن بنان
>  <list-general@mohsen.1.banan.byname.net>
> Date: Tue, 02 Aug 2011 20:04:24 -0700
> 
> 
> >>>>> On Tue, 02 Aug 2011 22:44:16 +0300, Eli Zaretskii <eliz@gnu.org> said:
> 
>   Eli> To fix the alignment, you need to append the LRM to each subject:
> 
>   Eli> :!   2 moons in the sky , August the 27th, يک آسمان و دو ماه‎|  37 |First LAST           |Jul 13 '10 
>   Eli> :!   Fw: بسیار جالب و خواندنی است ... فقط با دقت و حوصله بخو‎| 284 |First LAST           |14:15 
>   Eli> :!   Re: Next Saturday instead of this weekend             ‎|  36 |First LAST           |14:22 
> 
>   Eli> See how the line counts are nicely aligned now?  I inserted an LRM
>   Eli> just before the `|' separator to the left of the line count.  I did
>   Eli> that in the last line as well, but there it doesn't matter because the
>   Eli> subject uses only L2R text.
> 
>   Eli> To summarize, Gnus should append the LRM to each and every subject it
>   Eli> puts in the summary buffer.
> 
> The same thing applies to the From Name.
> First LAST in the above example.

But that's only for the possible customization of the summary format,
right?  With the format used in the above example, there's no problem,
right?

Re: bidi-display-reordering is now non-nil by default

[main]

>>>>> On Wed, 03 Aug 2011 04:42:37 -0400, Eli Zaretskii <eliz@gnu.org> said:

  >> From: محسن بنان
  >> <list-general@mohsen.1.banan.byname.net>
  >> Cc: Lars Magne Ingebrigtsen <larsi@gnus.org>,  emacs-devel@gnu.org,
  >> محسن بنان
  >> <list-general@mohsen.1.banan.byname.net>
  >> Date: Tue, 02 Aug 2011 20:04:24 -0700
  >> 
  >> 
  >> >>>>> On Tue, 02 Aug 2011 22:44:16 +0300, Eli Zaretskii <eliz@gnu.org> said:
  >> 
  Eli> To fix the alignment, you need to append the LRM to each subject:
  >> 
  Eli> :!  2 moons in the sky , August the 27th, يک آسمان و دو ماه‎|
  Eli> 37 |First LAST |Jul 13 '10
  Eli> :!  Fw: بسیار جالب و خواندنی است ... فقط با دقت و حوصله بخو‎|
  Eli> 284 |First LAST |14:15
  Eli> :!  Re: Next Saturday instead of this weekend ‎| 36 |First
  Eli> LAST |14:22
  >> 
  Eli> See how the line counts are nicely aligned now?  I inserted an LRM
  Eli> just before the `|' separator to the left of the line count.  I did
  Eli> that in the last line as well, but there it doesn't matter because the
  Eli> subject uses only L2R text.
  >> 
  Eli> To summarize, Gnus should append the LRM to each and every subject it
  Eli> puts in the summary buffer.
  >> 
  >> The same thing applies to the From Name.
  >> First LAST in the above example.

  Eli> But that's only for the possible customization of the summary format,
  Eli> right?  With the format used in the above example, there's no problem,
  Eli> right?

Correct.

I am saying that any field that can be R2L should
have the LRM so that customization of the summary
format would work right.

Re: bidi-display-reordering is now non-nil by default

[Stefan Monnier]

>> Chong, Stefan: how about having a (customizable) list of buffer names
>> which should by default have left-to-right paragraph direction?

Bad idea.  Better tell people to

  (add-hook 'foo-mode-hook
            (lambda ()
              (set (make-local-variable 'bidi-paragraph-direction)
                   'left-to-right)))

Using buffer names is really ugly (I know we use them for
display-buffer-alist, but I'm not too thrilled about it).

> for a mode to set the variable `paragraph-start' for Emacs to DTRT?  A
> quick glance at the Mew source shows that it does set `paragraph-start',
> though I don't know if every line is treated like a paragraph.

In non-editing modes, paragraph-start is only set to make M-} and M-{ do
something useful, and advancing by a single line is not something very
useful, so I wouldn't bet on it.

We could have a new bidi-paragraph-start setting, but in the OP's case
(and all similar cases, of which there are many) the right fix is to
disable auto-detection of paragraph direction and force it to
left-to-right.  And I suspect this will account for more than 90% of the
cases where one would be tempted to change bidi-paragraph-start.
So let's first see how far we can go with bidi-paragraph-direction.


        Stefan

Re: bidi-display-reordering is now non-nil by default

[Andrey Paramonov]

 > I sometimes get prompted to choose a different encoding when I
 > send email responses to mail that looked pure ASCII. I'm
 > presented with a buffer that points to a character that doesn't
 > look "foreign" at all. If I care to investigate, it turns out
 > that character was an em-dash or fancy quotes, that just
 > looked "normal" due to the font. Am I confused? No. I just
 > select the encoding, usually the one suggested by Emacs as the
 > default, and that's it.

If the insertion of special symbols can make the behavior of "special" 
buffers (like dired, Gnus summary etc) robust/correct, I see no problem 
in doing so always, not only in presence of R2L text. The prompt 
described above can only appear if one doesn't use Unicode and wants to 
save a special buffer (like dired) to a file. The combination of such 
events seems extremely rare (although not improbable, of course).

However if the special direction symbols will start to leak into some 
"usual" buffers, like programming language/xml modes, that might become 
a problem. Now if an UTF-8 buffer contains only latin characters, it's 
also ASCII. This is very common for a program source to be ASCII-only. 
But if special symbols are added to it by Emacs, it's no more ASCII and 
there is no guarantee that is will then correctly be compiled by another 
program.

Just an opinion from a L2R user here.

Best wishes,
Andrey Paramonov

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> Date: Mon, 15 Aug 2011 12:04:25 +0400
> From: Andrey Paramonov <paramon@acdlabs.ru>
> 
> However if the special direction symbols will start to leak into some 
> "usual" buffers, like programming language/xml modes, that might become 
> a problem. Now if an UTF-8 buffer contains only latin characters, it's 
> also ASCII. This is very common for a program source to be ASCII-only. 
> But if special symbols are added to it by Emacs, it's no more ASCII and 
> there is no guarantee that is will then correctly be compiled by another 
> program.
> 
> Just an opinion from a L2R user here.

Thanks.

Emacs should not (and does not) insert directional controls into any
"usual" buffers, unless the user inserts them manually, or asks for
that in some other way.  These control characters can "leak" into such
buffers only if the user copy/pastes text from some "unusual" buffer.
But the same issue will happen if one pastes text with some other
non-ASCII character, like Cyrillic.

Re: bidi-display-reordering is now non-nil by default

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Mon, 15 Aug 2011 12:04:25 +0400
>> From: Andrey Paramonov <paramon@acdlabs.ru>
>> 
>> However if the special direction symbols will start to leak into some 
>> "usual" buffers, like programming language/xml modes, that might become 
>> a problem. Now if an UTF-8 buffer contains only latin characters, it's 
>> also ASCII. This is very common for a program source to be ASCII-only. 
>> But if special symbols are added to it by Emacs, it's no more ASCII and 
>> there is no guarantee that is will then correctly be compiled by another 
>> program.
>> 
>> Just an opinion from a L2R user here.
>
> Thanks.
>
> Emacs should not (and does not) insert directional controls into any
> "usual" buffers, unless the user inserts them manually, or asks for
> that in some other way.  These control characters can "leak" into such
> buffers only if the user copy/pastes text from some "unusual" buffer.
> But the same issue will happen if one pastes text with some other
> non-ASCII character, like Cyrillic.

I can still pick iso-8869-5 encoding for a cyrillic cut&paste, but not
if it contains L2R marks.  And when having pure ASCII with L2R mark
leakage into it, I can't save under any 8-bit encoding (well, iso-8859-8
would work, but who is going to be able to read it?).

Emacs should really avoid formatting things with L2R marks that are not
actually required.

Is there a reason nobody responded to my repeated proposal to let
`format' deal with inserting L2R marks?

-- 
David Kastrup

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Mon, 15 Aug 2011 11:24:04 +0200
> 
> I can still pick iso-8869-5 encoding for a cyrillic cut&paste, but not
> if it contains L2R marks.

How is having Cyrillic characters different from having an LRM?

> And when having pure ASCII with L2R mark leakage into it, I can't
> save under any 8-bit encoding (well, iso-8859-8 would work, but who
> is going to be able to read it?).

ASCII characters are not changed by iso-8859-8 in any way, and LRM
doesn't need to be "read" at all.

> Emacs should really avoid formatting things with L2R marks that are not
> actually required.

You are welcome to code a function that determines when it is actually
required.  One problem with doing so with 100% accuracy is that the
outcome depends on the text that follows the string in question, and
sometimes even on text that precedes it.  These are not always known,
and in fact the text that follows will normally be unknown.

Re: bidi-display-reordering is now non-nil by default

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>
>> Emacs should really avoid formatting things with L2R marks that are not
>> actually required.
>
> You are welcome to code a function that determines when it is actually
> required.  One problem with doing so with 100% accuracy is that the
> outcome depends on the text that follows the string in question, and
> sometimes even on text that precedes it.  These are not always known,
> and in fact the text that follows will normally be unknown.

Your stance is that it is ok to pepper text prepared by Emacs (and the
Emacs sources all across the board) with LRM characters as long as it
has a moderate chance of getting closer to 100% accuracy _when_ Hebrew
characters move into the payload.

But you don't get that accuracy anyway since it _only_ works for those
areas that are explicitly prepared for it, and external maintainers of
modes will not change all their code on the theoretic assumption that
somebody might get ugly results when injecting Hebrew.

We have been that down the "everybody will adapt if forced to properly"
road already with multibyte characters in 20.2 or so, when
(forward-char) was different from (goto-char (1+ (point))).  That
approach failed and was replaced with a different, less efficient solution.

Any solution that requires code review and adaptation everywhere in code
that works fine without R2L content is not going to work.

Blind activism is not going to solve the problem you perceive.

-- 
David Kastrup

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Mon, 15 Aug 2011 12:46:09 +0200
> 
> Your stance is that it is ok to pepper text prepared by Emacs (and the
> Emacs sources all across the board) with LRM characters as long as it
> has a moderate chance of getting closer to 100% accuracy _when_ Hebrew
> characters move into the payload.
> 
> But you don't get that accuracy anyway since it _only_ works for those
> areas that are explicitly prepared for it, and external maintainers of
> modes will not change all their code on the theoretic assumption that
> somebody might get ugly results when injecting Hebrew.
> 
> We have been that down the "everybody will adapt if forced to properly"
> road already with multibyte characters in 20.2 or so, when
> (forward-char) was different from (goto-char (1+ (point))).  That
> approach failed and was replaced with a different, less efficient solution.
> 
> Any solution that requires code review and adaptation everywhere in code
> that works fine without R2L content is not going to work.
> 
> Blind activism is not going to solve the problem you perceive.

You are dead wrong: I have no activism stance in this matter.  I
actually don't care.  My job was to give Emacs infrastructure upon
which bidi-aware features could be built.  That job is almost done,
modulo bugs that will show during the pretest.  The rest is the
responsibility of those unnamed "external maintainers of modes"--they
are free not to care about catering to the hundreds of millions of
readers of R2L scripts enough to adapt: it's their funeral.  I myself
rarely if at all use any R2L scripts in Emacs.

Everything I wrote in this thread is just advice, no less, no more.
Advice from someone who knows something about how reordering works,
and what can or cannot be easily solved or coded.  You can take it or
you can continue attacking it, be my guest any way.  But if you want
to convince me, you will have to come up with something more tangible
than just FUD.

Re: bidi-display-reordering is now non-nil by default

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

> You are dead wrong: I have no activism stance in this matter.  I
> actually don't care.  My job was to give Emacs infrastructure upon
> which bidi-aware features could be built.  That job is almost done,
> modulo bugs that will show during the pretest.

And that is an important job that makes Emacs usable for a new audience.
It won't make it pretty for the new audience, but Emacs is not really
pretty except for users comfortable with an English interface.

> The rest is the responsibility of those unnamed "external maintainers
> of modes"--they are free not to care about catering to the hundreds of
> millions of readers of R2L scripts enough to adapt: it's their
> funeral.

Nope, it is Emacs' funeral if one can't expect it to produce consistent
results without lots of changes all across third party code bases.

My position is that we should be quite conservative with changing its
behavior for pure L2R material, or requiring such changes from
third-party code authors.

> Everything I wrote in this thread is just advice, no less, no more.
> Advice from someone who knows something about how reordering works,
> and what can or cannot be easily solved or coded.  You can take it or
> you can continue attacking it, be my guest any way.  But if you want
> to convince me, you will have to come up with something more tangible
> than just FUD.

There is no way that I can make you acknowledge something you choose to
ignore.  But since you are not the only Emacs developer, raising the
issues may still make them register with others.

-- 
David Kastrup

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Mon, 15 Aug 2011 13:27:47 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > You are dead wrong: I have no activism stance in this matter.  I
> > actually don't care.  My job was to give Emacs infrastructure upon
> > which bidi-aware features could be built.  That job is almost done,
> > modulo bugs that will show during the pretest.
> 
> And that is an important job that makes Emacs usable for a new audience.
> It won't make it pretty for the new audience, but Emacs is not really
> pretty except for users comfortable with an English interface.

There's nothing in Emacs yet to present an interface that is not
English.  I actually considered coding a couple of features to make it
possible for the UI to be displayed right-to-left, but eventually
decided against it, because it didn't make sense to do that for Emacs
that doesn't support localized UI at all.  (There are comments in the
code and a TODO item for someone who will want to implement UI
direction control in the future.)

However, what we are talking about here is not the interface, it's the
ability to edit bidirectional text.  Typing non-English text in Emacs
is dead simple, for quite some time, as simple as in any other
application on the same platform.  It is as "pretty" as it gets.

> > The rest is the responsibility of those unnamed "external maintainers
> > of modes"--they are free not to care about catering to the hundreds of
> > millions of readers of R2L scripts enough to adapt: it's their
> > funeral.
> 
> Nope, it is Emacs' funeral if one can't expect it to produce consistent
> results without lots of changes all across third party code bases.

If you think support for bidirectional scripts can come without "lots
of changes" where it matters, you are in the pipe dream land.  Sorry,
that's life.  Look how any bidi-aware HTML needs to use the equivalent
directives in order to DTRT--there's a lesson here to be learned.

> My position is that we should be quite conservative with changing its
> behavior for pure L2R material, or requiring such changes from
> third-party code authors.

If we can do that, fine.  The design and implementation of the bidi
infrastructure followed this principle from day one.  But sometimes,
there's nothing you can do except change the code in higher levels.
After all, breaking the assumption that "before" in the buffer means
"to the left" on the screen is an extremely fundamental change.  Any
code that depends on that assumption will need to do _something_ to
produce legible display.

Now, if someone can show a way to fix this use case in Emacs
infrastructure such as `format', I'm all ears.  You raised the idea,
but never explained it in enough details to judge it.  However, from
what I gather, your suggestion means changes in 3rd party code anyway
(e.g., use a special format specifier).

> There is no way that I can make you acknowledge something you choose
> to ignore.

Your views are not ignored, they were carefully considered and
rejected.

> But since you are not the only Emacs developer, raising the issues
> may still make them register with others.

I would be more than thrilled to see someone to come on board and
help.

Re: bidi-display-reordering is now non-nil by default

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

> However, what we are talking about here is not the interface, it's the
> ability to edit bidirectional text.

It would make discussion more productive if you actually remembered what
you are getting defensive about.

The point of discussion was the display (and preparation) of mode lines
and prompts as one example of Emacs-prepared text, going to special
buffer displays like dired, and whether we want Emacs to insert
directional marks in content like that even when no R2L material is
present.

The ability to edit bidirectional text is not affected either way by
Emacs inserting directional marks into strings it generates.

>> But since you are not the only Emacs developer, raising the issues
>> may still make them register with others.
>
> I would be more than thrilled to see someone to come on board and
> help.

Making Emacs annoying to L2R users might be one way to capture their
attention.  I can't deny it worked for multibyte support.  But I am not
fond of that approach.

-- 
David Kastrup

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Mon, 15 Aug 2011 14:56:50 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > However, what we are talking about here is not the interface, it's the
> > ability to edit bidirectional text.
> 
> It would make discussion more productive if you actually remembered what
> you are getting defensive about.

Sorry, my memory is limited.  This has been a very long thread.

> The point of discussion was the display (and preparation) of mode lines
> and prompts as one example of Emacs-prepared text, going to special
> buffer displays like dired

The main issue that started this was the Gnus summary and similar
buffers, which present table-like summary of a list of items.  Dired
was barely mentioned, and I don't remember prompts or mode lines
mentioned, except as in passage.  Please remind me what was the
problem with the mode line.

> and whether we want Emacs to insert directional marks in content
> like that even when no R2L material is present.

Yes.  To reiterate, I think doing this unconditionally is the easiest
way, since any kind of conditions will not be able to be 100% accurate
without lots of complications for the users of the relevant APIs.
It's quite possible that 100% accuracy will be entirely impractical.

> Making Emacs annoying to L2R users might be one way to capture their
> attention.  I can't deny it worked for multibyte support.  But I am not
> fond of that approach.

I have no idea what is this about.

Re: bidi-display-reordering is now non-nil by default

[Stefan Monnier]

> Emacs should really avoid formatting things with L2R marks that are not
> actually required.

And then

> Is there a reason nobody responded to my repeated proposal to let
> `format' deal with inserting L2R marks?

AFAICT these two are contradictory.  To only insert them when they are
really needed, we need additional hints that only a human can give.
Hence the new function.  Doing it in format would risk doing it in many
more cases where they're not actually indispensable.

But in any case, the bidi support in Emacs-24.1 is just a first step.
Clearly we'll need more work to make it work well.  The most obvious
case being R2L strings within programming and markup languages: adding
LRM marks in them would be needed to have them display correctly, but
at the same time we really don't want to add those LRM marks in the
corresponding file (nor in copy&paste text).

I think arguing about how best to solve problems in gnus-sum is futile
until we've figured out how to solve the problem for
markup&programming modes.
And in the mean time, using string-mark-left-to-right seems like
a fine stopgap.


        Stefan

Re: bidi-display-reordering is now non-nil by default

[David Kastrup]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Emacs should really avoid formatting things with L2R marks that are not
>> actually required.
>
> And then
>
>> Is there a reason nobody responded to my repeated proposal to let
>> `format' deal with inserting L2R marks?
>
> AFAICT these two are contradictory.

If "format" creates a pure L2R string, it will not insert directional
marks.  That is rather simple.

When it puts stuff together that would make cause changes of
directionality from inside a field to outside, it would insert marks to
avoid this.  So format("%sx", v) would not be equivalent to concat(v,
"x").  Something like format("%\\sx", v) might: one would try to make
some modifier for "don't insert directional marks".

> To only insert them when they are really needed, we need additional
> hints that only a human can give.

To get a hit rate of 100%, sure, nothing but human intervention will do.
But if the point is to get behavior that is rather close to optimal
without regressions for L2R texts and without human intervention
everywhere, addressing format is going to give a good start.  Where is
the sense in fields like %-10.10s" if format is going to cut off a
directional mark the user added for proper display?  If format is the
entity inserting the mark in the first place, at least this will go less
wrong.

> But in any case, the bidi support in Emacs-24.1 is just a first step.
> Clearly we'll need more work to make it work well.

Nothing wrong with that.  I'd like to avoid a situation where we'll need
to backpaddle to make it work well.

-- 
David Kastrup

Re: bidi-display-reordering is now non-nil by default

[Stefan Monnier]

> Nothing wrong with that.  I'd like to avoid a situation where we'll need
> to backpaddle to make it work well.

The only special thing we have currently installed is
string-mark-left-to-right which is not only a trivial functionality
that's easy to support in the future, but whose uses are also trivial to
find if we come up with something better in the future.

So, I think it will do fine.  Feel free to experiment with a patch to
`format' which we might even include in 24.2, but that's too risky at
this point.


        Stefan

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Mon, 15 Aug 2011 16:18:14 +0200
> 
> If "format" creates a pure L2R string, it will not insert directional
> marks.

What is a "pure L2R string", for this purpose?  Is this a string that
does not get reordered for display, or a string that does not affect
strings concatenated with it?  Or something else?  I think this should
be carefully defined to make sense in this context; each of the above
alternatives could yield a different result for the same string.

> When it puts stuff together that would make cause changes of
> directionality from inside a field to outside, it would insert marks to
> avoid this.  So format("%sx", v) would not be equivalent to concat(v,
> "x").  Something like format("%\\sx", v) might: one would try to make
> some modifier for "don't insert directional marks".

This might work (although is non-trivial to code), but it requires
that the following string(s) be known to `format'.  Note: not just the
single following substring, but as much text as is needed to determine
whether a simple concatenation will or will not work.  For example, if
"x" in the above example is a string made of weak or neutral
characters (e.g., digits or punctuation or whitespace), `format' will
be unable to DTRT, unless we accept the sub-optimal solution of
inserting directional controls in this case anyway.

Of course, it also means relevant Lisp code will have to be modified
to not use `concat' or `insert' to generate text out of its fragments,
but instead to use these special features of `format'.

> But if the point is to get behavior that is rather close to optimal
> without regressions for L2R texts

This "rather close to optimal" goal should also be clearly defined,
IMO.

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Mon, 15 Aug 2011 09:59:34 -0400
> Cc: emacs-devel@gnu.org
> 
> I think arguing about how best to solve problems in gnus-sum is futile
> until we've figured out how to solve the problem for
> markup&programming modes.

I don't think so.  The former is a much simpler problem, because it
has to do with text that normally isn't edited nor submitted to
external programs.  It is basically a problem of concatenating strings
subject to certain restrictions wrt their layout on the screen, like
"make str1 display to the left of str2".

The latter problem is much harder, because it needs a capability to
reorder only certain portions of a buffer.

Re: bidi-display-reordering is now non-nil by default

[Stefan Monnier]

>> I think arguing about how best to solve problems in gnus-sum is futile
>> until we've figured out how to solve the problem for
>> markup&programming modes.
> I don't think so.  The former is a much simpler problem, because it
> has to do with text that normally isn't edited nor submitted to
> external programs.  It is basically a problem of concatenating strings
> subject to certain restrictions wrt their layout on the screen, like
> "make str1 display to the left of str2".
> The latter problem is much harder, because it needs a capability to
> reorder only certain portions of a buffer.

We agree.  I just think that once we've solved the hard problem, the
best solution for the simple problem will probably be quite different
from what we would have come up with at first.

And currently string-mark-left-to-right does the job just fine.


        Stefan

Re: bidi-display-reordering is now non-nil by default

[Lars Magne Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> And currently string-mark-left-to-right does the job just fine.

I'm catching up on mail, so I might be responding to something that has
already been decided otherwise, but since `string-mark-left-to-right'
still seems to exist in the sources, I'm guessing it's still the
preferred solution.  :-)

Looking at the code, it just appends "\x200e" to the end of r2l strings,
right?  This means that for Gnus to just bind `gnus-tmp-subject-or-nil'
to `(string-mark-left-to-right gnus-tmp-subject-or-nil)' won't work --
the subjects are liable to be chopped off by length restriction specs.
So this will probably have to be special-cased somewhere in the
format-spec machinery, which should be possible without wreaking too
much havoc.

But it's unlikely that this will work properly for people who have
already customised their `gnus-summary-format-line', since that may have
to change.  I'll see what I can come up with...

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Mon, 15 Aug 2011 14:13:21 -0400
> Cc: dak@gnu.org, emacs-devel@gnu.org
> 
> And currently string-mark-left-to-right does the job just fine.

In line with the above, does the following patch look like TRT?  It is
needed, e.g., when displaying completions for C-x C-f when some of the
files in the directory end in R2L characters.  Since we cover the TAB
with a `(display (space :align-to ...))' property, which makes it a
``replacing'' display property, the underlying TAB loses its
bidirectional properties and is reordered as if it were a neutral
character.  So instead of

  OOF                RAB

I see

  RAB                OOF

which shows the candidates in the wrong order, especially if
completions-format is `vertical'.

If this patch is okay, can you tell whether other completion
facilities in Emacs might need similar changes?

=== modified file 'lisp/minibuffer.el'
--- lisp/minibuffer.el	2011-08-15 16:10:39 +0000
+++ lisp/minibuffer.el	2011-08-18 13:07:52 +0000
@@ -1119,13 +1119,24 @@ It also eliminates runs of equal strings
 				       `(display (space :align-to ,column)))
 		  nil))))
             (if (not (consp str))
-                (put-text-property (point) (progn (insert str) (point))
+                (put-text-property (point)
+				   (progn
+				     (insert (bidi-string-mark-left-to-right str))
+				     (point))
                                    'mouse-face 'highlight)
-              (put-text-property (point) (progn (insert (car str)) (point))
+              (put-text-property (point)
+				 (progn
+				   (insert
+				    (bidi-string-mark-left-to-right (car str)))
+				   (point))
                                  'mouse-face 'highlight)
-              (add-text-properties (point) (progn (insert (cadr str)) (point))
+              (add-text-properties (point)
+				   (progn
+				     (insert
+				      (bidi-string-mark-left-to-right (cadr str)))
+				     (point))
                                    '(mouse-face nil
-                                     face completions-annotations)))
+						face completions-annotations)))
 	    (cond
 	     ((eq completions-format 'vertical)
 	      ;; Vertical format

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

Ping!

> Date: Thu, 18 Aug 2011 19:14:38 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> > From: Stefan Monnier <monnier@iro.umontreal.ca>
> > Date: Mon, 15 Aug 2011 14:13:21 -0400
> > Cc: dak@gnu.org, emacs-devel@gnu.org
> > 
> > And currently string-mark-left-to-right does the job just fine.
> 
> In line with the above, does the following patch look like TRT?  It is
> needed, e.g., when displaying completions for C-x C-f when some of the
> files in the directory end in R2L characters.  Since we cover the TAB
> with a `(display (space :align-to ...))' property, which makes it a
> ``replacing'' display property, the underlying TAB loses its
> bidirectional properties and is reordered as if it were a neutral
> character.  So instead of
> 
>   OOF                RAB
> 
> I see
> 
>   RAB                OOF
> 
> which shows the candidates in the wrong order, especially if
> completions-format is `vertical'.
> 
> If this patch is okay, can you tell whether other completion
> facilities in Emacs might need similar changes?
> 
> === modified file 'lisp/minibuffer.el'
> --- lisp/minibuffer.el	2011-08-15 16:10:39 +0000
> +++ lisp/minibuffer.el	2011-08-18 13:07:52 +0000
> @@ -1119,13 +1119,24 @@ It also eliminates runs of equal strings
>  				       `(display (space :align-to ,column)))
>  		  nil))))
>              (if (not (consp str))
> -                (put-text-property (point) (progn (insert str) (point))
> +                (put-text-property (point)
> +				   (progn
> +				     (insert (bidi-string-mark-left-to-right str))
> +				     (point))
>                                     'mouse-face 'highlight)
> -              (put-text-property (point) (progn (insert (car str)) (point))
> +              (put-text-property (point)
> +				 (progn
> +				   (insert
> +				    (bidi-string-mark-left-to-right (car str)))
> +				   (point))
>                                   'mouse-face 'highlight)
> -              (add-text-properties (point) (progn (insert (cadr str)) (point))
> +              (add-text-properties (point)
> +				   (progn
> +				     (insert
> +				      (bidi-string-mark-left-to-right (cadr str)))
> +				     (point))
>                                     '(mouse-face nil
> -                                     face completions-annotations)))
> +						face completions-annotations)))
>  	    (cond
>  	     ((eq completions-format 'vertical)
>  	      ;; Vertical format
> 
> 
> 

Re: bidi-display-reordering is now non-nil by default

[Stefan Monnier]

>> files in the directory end in R2L characters.  Since we cover the TAB
>> with a `(display (space :align-to ...))' property, which makes it a
>> ``replacing'' display property, the underlying TAB loses its
>> bidirectional properties and is reordered as if it were a neutral
>> character.  So instead of

I guess this brings us back to "a way to mark some char as a field
separator, just like a TAB; and in this particular case it clearly would
be fine to do it via a `display' property.

Some might even argue that a (space :align-to ...) display property is
sufficiently similar to a TAB that such a property should be interpreted
similarly to a field separator by the bidi reordering code.

>> If this patch is okay,

Assuming it's not straightforward to change the C code to handle such
display properties (not simple enough for 24.1, or maybe we're not sure
it's actually a good idea to do it), then your patch looks like an
OK solution.

>> can you tell whether other completion facilities in Emacs might need
>> similar changes?

I'd tend to think that most/all other completion facilities should be
fixed by using the generic code rather than by fixing their code, so
they shouldn't need similar changes.


        Stefan

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Mon, 22 Aug 2011 15:35:24 -0400
> 
> I guess this brings us back to "a way to mark some char as a field
> separator, just like a TAB; and in this particular case it clearly would
> be fine to do it via a `display' property.
> 
> Some might even argue that a (space :align-to ...) display property is
> sufficiently similar to a TAB that such a property should be interpreted
> similarly to a field separator by the bidi reordering code.

Only :align-to, or any other properties supported by the `space'
display spec?  If only the former, why only that?  Why not :width, for
example?

> Assuming it's not straightforward to change the C code to handle such
> display properties (not simple enough for 24.1, or maybe we're not sure
> it's actually a good idea to do it)

It wouldn't be hard to add this feature, if you think it's okay to do
that now, the feature freeze notwithstanding.  I would need the answer
to the question above, though.

> >> can you tell whether other completion facilities in Emacs might need
> >> similar changes?
> 
> I'd tend to think that most/all other completion facilities should be
> fixed by using the generic code rather than by fixing their code, so
> they shouldn't need similar changes.

"Generic code" meaning treating a `space' display spec as a segment
separator?  If that's not what you meant, I don't understand what
other generic solution would be possible.

In any case, please humor me by giving the list of completion packages
outside of minibuffer.el, as my knowledge of this area barely covers
the standard completion facilities.

Re: bidi-display-reordering is now non-nil by default

[Stefan Monnier]

>> I guess this brings us back to "a way to mark some char as a field
>> separator, just like a TAB; and in this particular case it clearly would
>> be fine to do it via a `display' property.
>> Some might even argue that a (space :align-to ...) display property is
>> sufficiently similar to a TAB that such a property should be interpreted
>> similarly to a field separator by the bidi reordering code.
> Only :align-to, or any other properties supported by the `space'
> display spec?

Good question.

> If only the former, why only that?

At least the :align-to has a clear similarity to the TAB char.

> Why not :width, for example?

I wouldn't know.  That's the problem.  I guess it would be a heuristic
in any case, so we'd want to provide some way to control the behavior in
case the heuristic is wrong.

>> Assuming it's not straightforward to change the C code to handle such
>> display properties (not simple enough for 24.1, or maybe we're not sure
>> it's actually a good idea to do it)
> It wouldn't be hard to add this feature, if you think it's okay to do
> that now, the feature freeze notwithstanding.  I would need the answer
> to the question above, though.

Maybe all `space' display properties should be treated as field
separators, but with a new :not-a-bidi-field-separator property
that can override this default.

Not sure which way the default should go, since OT1H I tend to feel like
the default should be to treat such spaces as field separators, based on
their similarity to TAB and based on the current way they're used in
tabulated data, but OTOH maybe this is a reflection of the lack of R2L
support until now.

>> >> can you tell whether other completion facilities in Emacs might need
>> >> similar changes?
>> I'd tend to think that most/all other completion facilities should be
>> fixed by using the generic code rather than by fixing their code, so
>> they shouldn't need similar changes.
> "Generic code" meaning treating a `space' display spec as a segment
> separator?

No, "generic code" meaning the "built-in" completion code in
minibuffer.el, as opposed to alternative completion packages that may
re-implement all or part of the completion code (ido, iswitchb,
auto-complete, company-mode, ...).

> In any case, please humor me by giving the list of completion packages
> outside of minibuffer.el, as my knowledge of this area barely covers
> the standard completion facilities.

For Emacs-24, I've made an effort to try and reduce the amount of
alternative completion code (most of which was not like
ido/iswitchb/.. but more like lisp-mode or makefile-mode
re-implementing the simple completion code already implemented for the
minibuffer), so I guess that nowadays there's mostly just ido and iswitchb
bundled with Emacs plus company-mode in the GNU ELPA (hopefully soon
accompanied with auto-complete and completion-ui).


        Stefan

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Tue, 23 Aug 2011 14:19:37 -0400
> 
> >> I guess this brings us back to "a way to mark some char as a field
> >> separator, just like a TAB; and in this particular case it clearly would
> >> be fine to do it via a `display' property.
> >> Some might even argue that a (space :align-to ...) display property is
> >> sufficiently similar to a TAB that such a property should be interpreted
> >> similarly to a field separator by the bidi reordering code.
> > Only :align-to, or any other properties supported by the `space'
> > display spec?
> 
> Good question.
> 
> > If only the former, why only that?
> 
> At least the :align-to has a clear similarity to the TAB char.

I think they all do.  And the implementation is the same: we produce
white space of the specified dimensions.

Btw, I've looked at all the users of (space :SOMETHING) spec in
Emacs.  Almost all of them use :align-to.  The rest all use :width,
and with a single exception (ruler-mode, which uses this on the
fringes, and so is irrelevant to this discussion), all of them use
this to separate and align fields.

> Maybe all `space' display properties should be treated as field
> separators, but with a new :not-a-bidi-field-separator property
> that can override this default.

If we want to keep this simple enough to get into 24.1, I suggest to
treat them all as segment separators, and not introduce any overriding
properties until someone comes with a convincing use case that
actually needs something like that.

> > In any case, please humor me by giving the list of completion packages
> > outside of minibuffer.el, as my knowledge of this area barely covers
> > the standard completion facilities.
> 
> For Emacs-24, I've made an effort to try and reduce the amount of
> alternative completion code (most of which was not like
> ido/iswitchb/.. but more like lisp-mode or makefile-mode
> re-implementing the simple completion code already implemented for the
> minibuffer), so I guess that nowadays there's mostly just ido and iswitchb
> bundled with Emacs plus company-mode in the GNU ELPA (hopefully soon
> accompanied with auto-complete and completion-ui).

Thanks, I will take a look.

Need your decision on the space spec.

Re: bidi-display-reordering is now non-nil by default

[Stefan Monnier]

> Btw, I've looked at all the users of (space :SOMETHING) spec in
> Emacs.  Almost all of them use :align-to.  The rest all use :width,
> and with a single exception (ruler-mode, which uses this on the
> fringes, and so is irrelevant to this discussion), all of them use
> this to separate and align fields.

Thanks for looking at it.  It confirms my impression.

>> Maybe all `space' display properties should be treated as field
>> separators, but with a new :not-a-bidi-field-separator property
>> that can override this default.
> If we want to keep this simple enough to get into 24.1, I suggest to
> treat them all as segment separators, and not introduce any overriding
> properties until someone comes with a convincing use case that
> actually needs something like that.

OK.

>> For Emacs-24, I've made an effort to try and reduce the amount of
>> alternative completion code (most of which was not like
>> ido/iswitchb/.. but more like lisp-mode or makefile-mode
>> re-implementing the simple completion code already implemented for the
>> minibuffer), so I guess that nowadays there's mostly just ido and iswitchb
>> bundled with Emacs plus company-mode in the GNU ELPA (hopefully soon
>> accompanied with auto-complete and completion-ui).

> Thanks, I will take a look.

iswitchb already uses display-completion-list when displaying
completions in *Completions*.  And ido.el never displays anything in
*Completions*, AFAICT.
But both of them (along with icomplete.el) show a list of completions
"inline" in the minibuffer as in

  Switch to buffer: a{data.tex,foo.ada,*scratch*}

so maybe the {,,,} list should use bidi-string-mark-left-to-right before
the comma.  BTW, I think we should make it clear that
bidi-string-mark-left-to-right only makes sense in text that gets
displayed in L2R paragraphs, so it's largely restricted to buffers that
should set bidi-paragraph-direction to `left-to-right'.

> Need your decision on the space spec.

Is the above enough, or are there still more open issues?


        Stefan

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Tue, 23 Aug 2011 15:17:29 -0400
> 
> iswitchb already uses display-completion-list when displaying
> completions in *Completions*.  And ido.el never displays anything in
> *Completions*, AFAICT.
> But both of them (along with icomplete.el) show a list of completions
> "inline" in the minibuffer as in
> 
>   Switch to buffer: a{data.tex,foo.ada,*scratch*}
> 
> so maybe the {,,,} list should use bidi-string-mark-left-to-right before
> the comma.

I will look at how they do this.

> BTW, I think we should make it clear that
> bidi-string-mark-left-to-right only makes sense in text that gets
> displayed in L2R paragraphs, so it's largely restricted to buffers that
> should set bidi-paragraph-direction to `left-to-right'.

The ELisp manual already says that.

> > Need your decision on the space spec.
> 
> Is the above enough, or are there still more open issues?

If the above is a "go ahead" for 24.1, then no open issues left.

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Tue, 23 Aug 2011 15:17:29 -0400
> 
>   Switch to buffer: a{data.tex,foo.ada,*scratch*}
> 
> so maybe the {,,,} list should use bidi-string-mark-left-to-right before
> the comma.

They should probably run each completion candidate through
bidi-string-mark-left-to-right, yes.

I say "probably" because there's a separate but related issue with
buffer and file names that include R2L characters and end in weak or
neutral characters: those weak or neutrals will be displayed to the
left of the R2L characters, as in "DCBA!" (instead of the more
plausible "!DCBA"), and appending the LRM will not cure that.  So
perhaps we should have a more comprehensive plan for handling these
issues before we start sprinkling bidi-string-mark-left-to-right all
over the place.

It is also possible we should add a more general bidi-string-mark-dwim
function, which uses either LRM or RLM depending on the paragraph
direction, to avoid additional changes later, when the UI will be
localizable.

Re: bidi-display-reordering is now non-nil by default

[Stefan Monnier]

>> Switch to buffer: a{data.tex,foo.ada,*scratch*}
>> so maybe the {,,,} list should use bidi-string-mark-left-to-right before
>> the comma.
> They should probably run each completion candidate through
> bidi-string-mark-left-to-right, yes.

That was my impression as well.

> I say "probably" because there's a separate but related issue with
> buffer and file names that include R2L characters and end in weak or
> neutral characters: those weak or neutrals will be displayed to the
> left of the R2L characters, as in "DCBA!" (instead of the more
> plausible "!DCBA"), and appending the LRM will not cure that.

I'm a bit confused here: you say "to the left of the R2L characters",
but your example shows it to the right.

> So perhaps we should have a more comprehensive plan for handling these
> issues before we start sprinkling bidi-string-mark-left-to-right all
> over the place.

Sounds fine to me, yes.  I don't think there's any hurry to try and
address all these issues before 24.1.

This said, it seems the problem you're referring to is a lot more
general than to icomplete/ido/iswitchb: the same problem will show up
wherever the file/buffer is displayed, no?

> It is also possible we should add a more general bidi-string-mark-dwim
> function, which uses either LRM or RLM depending on the paragraph
> direction, to avoid additional changes later, when the UI will be
> localizable.

Yes, that's what I was getting at, basically I think that we almost
never want to add a LRM but instead we want to add some kind of
segment-separator.
E.g. I think that a buffer ABC<1> should be either displayed as CBA<1>
or <1>CBA depending on the surrounding text direction (which might be
different from the paragraph direction).

> > > Need your decision on the space spec.
> > Is the above enough, or are there still more open issues?
> If the above is a "go ahead" for 24.1, then no open issues left.

Yes, it's a "go ahead" for the "treat any `space' display property as
a segment separator".


        Stefan

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Wed, 24 Aug 2011 10:51:34 -0400
> 
> >> Switch to buffer: a{data.tex,foo.ada,*scratch*}
> >> so maybe the {,,,} list should use bidi-string-mark-left-to-right before
> >> the comma.
> > They should probably run each completion candidate through
> > bidi-string-mark-left-to-right, yes.
> 
> That was my impression as well.
> 
> > I say "probably" because there's a separate but related issue with
> > buffer and file names that include R2L characters and end in weak or
> > neutral characters: those weak or neutrals will be displayed to the
> > left of the R2L characters, as in "DCBA!" (instead of the more
> > plausible "!DCBA"), and appending the LRM will not cure that.
> 
> I'm a bit confused here: you say "to the left of the R2L characters",
> but your example shows it to the right.

Yes, I meant the other left ;-)

> This said, it seems the problem you're referring to is a lot more
> general than to icomplete/ido/iswitchb: the same problem will show up
> wherever the file/buffer is displayed, no?

Yes, but with buffer/file contents it's a different story.  Whoever
writes the buffer/file text can (and should) use the directional
controls or accept the consequences.

By contrast, what Emacs shows in the echo area, the minibuffer, and
the buffers it pops up, like *Completions*, is something Emacs itself
generates, and the user has no control on those contents.  So we
should have some general policy in place, and have supporting
infrastructure, that would allow a Lisp program to DTRT with names of
files, buffers, etc. that we show in these contexts.

For example, when we prompt for something and show the default value,
the presence of R2L characters in that default should not produce the
kind of jumbled display we sometimes show with the current codebase.
Here's a simple example:

 emacs -Q
 M-: (read-buffer "Buffer name: " "אבגדה") RET
 M-n

> > It is also possible we should add a more general bidi-string-mark-dwim
> > function, which uses either LRM or RLM depending on the paragraph
> > direction, to avoid additional changes later, when the UI will be
> > localizable.
> 
> Yes, that's what I was getting at, basically I think that we almost
> never want to add a LRM but instead we want to add some kind of
> segment-separator.

It turns out a segment separator (in the UAX#9 definition, i.e. a TAB)
does not always help.  Here, for example, it does not, because the
colon and the closing parenthesis interfere:

     Buffer name (default אבגד):	12345

Remove the colon and the closing paren, and everything is dandy:

     Buffer name (default אבגד	12345

This is a strange behavior, it caught me by surprise.  But this
behavior is correct, as confirmed by Yudit and by the UAX#9 Reference
Implementation, so it's not a bug.  It's what the UBA mandates.

It looks like the only solution that always works is to use the
Paragraph Separator character (u+2029).  That's because each paragraph
is reordered separately and independently.

> E.g. I think that a buffer ABC<1> should be either displayed as CBA<1>
> or <1>CBA depending on the surrounding text direction (which might be
> different from the paragraph direction).

Not sure what you mean here.  Do you mean reorder that as a single
atomic unit?  If not, what do you mean by "surrounding text
direction"?  The surrounding text is different for each character of
the "<1>" string, and what we do with it by default is precisely
display depending on the surrounding text direction.

> > > > Need your decision on the space spec.
> > > Is the above enough, or are there still more open issues?
> > If the above is a "go ahead" for 24.1, then no open issues left.
> 
> Yes, it's a "go ahead" for the "treat any `space' display property as
> a segment separator".

Well, in light of the above, it sounds like we need to treat such
properties as a paragraph separator, to get the effect we need.  I'll
take a stab at that.

Re: bidi-display-reordering is now non-nil by default

[Stefan Monnier]

>> > I say "probably" because there's a separate but related issue with
>> > buffer and file names that include R2L characters and end in weak or
>> > neutral characters: those weak or neutrals will be displayed to the
>> > left of the R2L characters, as in "DCBA!" (instead of the more
>> > plausible "!DCBA"), and appending the LRM will not cure that.
>> I'm a bit confused here: you say "to the left of the R2L characters",
>> but your example shows it to the right.
> Yes, I meant the other left ;-)
>> This said, it seems the problem you're referring to is a lot more
>> general than to icomplete/ido/iswitchb: the same problem will show up
>> wherever the file/buffer is displayed, no?
> Yes, but with buffer/file contents it's a different story.

I meant buffer/file *name*, sorry.
But I think I now understand the problem you're talking about:
ABCD! all on its own would be properly displayed as !DCBA, but if it is
followed by an LRM mark then it will turn into DCBA!.  So, yes, that's
a problem.  What about adding a TAB instead of an LRM?

>  emacs -Q
>  M-: (read-buffer "Buffer name: " "אבגדה") RET
>  M-n

Right.  This is one more example of a "field".  I think we'd want
a function like bidi-independent-string which takes a string STR and
returns a new string that displays just like STR does but with the added
twist that none of the surrounding text will ever get displayed in the
middle of STR and that text that comes after STR gets displayed "after"
and text that comes before gets displayed "before".
I'm not sure if that is possible or even meaningful, tho.

>> E.g. I think that a buffer ABC<1> should be either displayed as CBA<1>
>> or <1>CBA depending on the surrounding text direction (which might be
>> different from the paragraph direction).
> If not, what do you mean by "surrounding text direction"?

Whether this ABC<1> appears within an L2R chunk of text or an L2R chunk
of text.  I guess that "paragraph direction" is a good approximation.

>> Yes, it's a "go ahead" for the "treat any `space' display property as
>> a segment separator".
> Well, in light of the above, it sounds like we need to treat such
> properties as a paragraph separator, to get the effect we need.  I'll
> take a stab at that.

Thanks,


        Stefan

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Thu, 25 Aug 2011 00:38:51 -0400
> 
> But I think I now understand the problem you're talking about:
> ABCD! all on its own would be properly displayed as !DCBA, but if it is
> followed by an LRM mark then it will turn into DCBA!.  So, yes, that's
> a problem.  What about adding a TAB instead of an LRM?

Won't help, you need an RLM after the exclam.  Or enclose the entire
string in RLE..PDF, see below.

> >  emacs -Q
> >  M-: (read-buffer "Buffer name: " "אבגדה") RET
> >  M-n
> 
> Right.  This is one more example of a "field".  I think we'd want
> a function like bidi-independent-string which takes a string STR and
> returns a new string that displays just like STR does but with the added
> twist that none of the surrounding text will ever get displayed in the
> middle of STR and that text that comes after STR gets displayed "after"
> and text that comes before gets displayed "before".
> I'm not sure if that is possible or even meaningful, tho.

It's possible.  It's called "embedding".  Here (watch the directional
control characters around the parentheses):

   Buffer name ‪(default "אבגדה"):‬ אבגדרשת123

Should we provide bidi-embed-string-l2r/r2l functions?

> >> E.g. I think that a buffer ABC<1> should be either displayed as CBA<1>
> >> or <1>CBA depending on the surrounding text direction (which might be
> >> different from the paragraph direction).
> > If not, what do you mean by "surrounding text direction"?
> 
> Whether this ABC<1> appears within an L2R chunk of text or an L2R chunk
> of text.  I guess that "paragraph direction" is a good approximation.

Why not always display it is <1>CBA ?  That's the natural visual
order.

Re: bidi-display-reordering is now non-nil by default

[Stefan Monnier]

> It's possible.  It's called "embedding".  Here (watch the directional
> control characters around the parentheses):
>    Buffer name ‪(default "אבגדה"):‬ אבגדרשת123
> Should we provide bidi-embed-string-l2r/r2l functions?

That sounds like just what we need, then.  How 'bout providing
a (bidi-embed-string STR &optional DIRECTION) where DIRECTION works like
bidi-paragraph-direction (i.e. nil means auto-detect).

>> Whether this ABC<1> appears within an L2R chunk of text or an L2R chunk
>> of text.  I guess that "paragraph direction" is a good approximation.
> Why not always display it is <1>CBA ?  That's the natural visual order.

To me, in an L2R context, any buffer name should be displayed as
NAME<N>, regardless of whether NAME happens to be R2L or L2R (i.e. NAME
should be an "embedded-string").

> Done in revision 105562 on the trunk.  Please give it a try.  Initial
> testing with the *Completions* buffer indicates that it does TRT.
> I've also removed the calls to bidi-string-mark-left-to-right from
> buff-menu.el, as they are no longer needed there.

Great, thanks,


        Stefan

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Thu, 25 Aug 2011 23:55:42 -0400
> 
> > It's possible.  It's called "embedding".  Here (watch the directional
> > control characters around the parentheses):
> >    Buffer name ‪(default "אבגדה"):‬ אבגדרשת123
> > Should we provide bidi-embed-string-l2r/r2l functions?
> 
> That sounds like just what we need, then.  How 'bout providing
> a (bidi-embed-string STR &optional DIRECTION) where DIRECTION works like
> bidi-paragraph-direction (i.e. nil means auto-detect).

Auto-detect is not really well-defined in this case, because the
DWIM-ish value of DIRECTION is determined by 3 factors:

 . the direction of the paragraph where STR will be inserted
 . the text around the position where it will be inserted
 . the desired effect of embedding the string

As you see, nothing here depends on the string itself.  The first 2
factors could be perhaps taken care of by providing additional
arguments that tell where the string will be inserted, although in
many cases the surrounding text will not be known yet (think inserting
a prompt into the minibuffer), at least not on both ends.  But the 3rd
factor is a complete mystery at this point, as I don't yet have enough
experience with using embeddings to enumerate all the possible use
cases.  If someone else can help produce such a list, I'm all ears.
Failing that, this function will have to be used by "experts" or after
some trial and error, and DIRECTION will have to be non-optional.

Alternatively, we could for now support nil as DIRECTION by assuming
the string will be inserted in the current buffer at point and using
the paragraph direction there.  It seems to DTRT in the simple use
cases I tried.  We could also add an optional BUFFER argument to use
the information from that buffer.

I guess what I'm saying is that the API will almost certainly change
as we gain experience.  So I'm unsure how elaborate we should make it
at this point, because any mistake will mean more changes in the
future when the API changes.

> >> Whether this ABC<1> appears within an L2R chunk of text or an L2R chunk
> >> of text.  I guess that "paragraph direction" is a good approximation.
> > Why not always display it is <1>CBA ?  That's the natural visual order.
> 
> To me, in an L2R context, any buffer name should be displayed as
> NAME<N>, regardless of whether NAME happens to be R2L or L2R (i.e. NAME
> should be an "embedded-string").

We will get what you want if we embed NAME in LRE..PDF when it's in an
L2R paragraph, and in RLE..PDF when it's in an R2L paragraph.  Or if
we insert LRM/RLM between NAME and <n>.  The difficulty is that you
need to know in advance the paragraph direction where the string will
be inserted.  This means the embedding should happen very close to
actually inserting the string.  IOW, utility functions such as
buffer-name cannot magically return a correctly decorated buffer name
and, what's worse, application code will need to take NAME<n> apart,
embed NAME or add the mark to it, then catenate <n> to it -- perhaps
we should have a function to do that.

There's a similar issue with file names: /abcd/ABCD/XYZ will be
displayed as /abcd/ZYX/DCBA.

Btw, using LRM/RLM is always preferred when possible, because it's
stateless, as opposed to an embedding.  UAX#9 says that much.  Also,
more single-byte encodings support the marks than the rest of the
directional controls.

So a related question is: should we prefer the marks where possible,
or should we standardize on embeddings?  I'm not sure we are equipped
with enough experience to decide at this time.  However, an important
practical question is: which alternative to use to fix read_minibuf,
for example?

Maybe we should embed fields and use the marks elsewhere?

Re: bidi-display-reordering is now non-nil by default

[Stefan Monnier]

> I guess what I'm saying is that the API will almost certainly change
> as we gain experience.  So I'm unsure how elaborate we should make it
> at this point, because any mistake will mean more changes in the
> future when the API changes.

Right.  Better not solve this problem yet.

> we insert LRM/RLM between NAME and <n>.  The difficulty is that you
> need to know in advance the paragraph direction where the string will
> be inserted.  This means the embedding should happen very close to

Hmm... so UAX#9 does not give us the tools we need, here.

> There's a similar issue with file names: /abcd/ABCD/XYZ will be
> displayed as /abcd/ZYX/DCBA.

IIUC in an English paragraph this should be displayed as /abcd/DCBA/ZYX
but in a Hebrew paragraph it should be displayed as ZYX/DCBA/abcd/, right?

> Btw, using LRM/RLM is always preferred when possible, because it's
> stateless, as opposed to an embedding.  UAX#9 says that much.  Also,

Yes, that makes a lot of sense (just think of using substring to select
part of the embedding, dropping the initial or final marker).

So embeddings should not be used too much, and/or should only be used
dynamically (jit-lock-style) and implicitly (e.g. not copied by
kill-region).

> So a related question is: should we prefer the marks where possible,
> or should we standardize on embeddings?  I'm not sure we are equipped
> with enough experience to decide at this time.

Agreed.

> However, an important practical question is: which alternative to use
> to fix read_minibuf, for example?

I don't think it matters too much; just use whatever works best.


        Stefan

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: emacs-devel@gnu.org
> Date: Fri, 26 Aug 2011 22:53:36 -0400
> 
> > There's a similar issue with file names: /abcd/ABCD/XYZ will be
> > displayed as /abcd/ZYX/DCBA.
> 
> IIUC in an English paragraph this should be displayed as /abcd/DCBA/ZYX
> but in a Hebrew paragraph it should be displayed as ZYX/DCBA/abcd/, right?

I think it should be displayed as /abcd/DCBA/ZYX in both cases.  When
a file name is mentioned in a R2L paragraph, it will need to be
embedded in LRE..PDF, as I did with the URL near the end of TUTORIAL.he.

> So embeddings should not be used too much, and/or should only be used
> dynamically (jit-lock-style) and implicitly (e.g. not copied by
> kill-region).

It depends.  If only part of the embedded text is copied, then the
directional formats should be dropped, indeed.  But if all of it is
copied, they should be kept, to preserve the visual appearance.

So I guess we will need to set filter-buffer-substring-functions to
something non-nil by default?  Or maybe modify filter-buffer-substring
itself to remove these characters as needed?

Re: bidi-display-reordering is now non-nil by default

[Stefan Monnier]

>> So embeddings should not be used too much, and/or should only be used
>> dynamically (jit-lock-style) and implicitly (e.g. not copied by
>> kill-region).
> It depends.  If only part of the embedded text is copied, then the
> directional formats should be dropped, indeed.  But if all of it is
> copied, they should be kept, to preserve the visual appearance.

Sounds difficult to implement reliably.

> So I guess we will need to set filter-buffer-substring-functions to
> something non-nil by default?  Or maybe modify filter-buffer-substring
> itself to remove these characters as needed?

Maybe that can be made to work, but we'd need to add a corresponding
filter-substring, and I'm not sure that we wouldn't get into bugs where
we need to change the code from buffer-substring to
filter-buffer-substring.


        Stefan

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Sat, 27 Aug 2011 22:52:00 -0400
> 
> >> So embeddings should not be used too much, and/or should only be used
> >> dynamically (jit-lock-style) and implicitly (e.g. not copied by
> >> kill-region).
> > It depends.  If only part of the embedded text is copied, then the
> > directional formats should be dropped, indeed.  But if all of it is
> > copied, they should be kept, to preserve the visual appearance.
> 
> Sounds difficult to implement reliably.

What difficulties do you have in mind?

> > So I guess we will need to set filter-buffer-substring-functions to
> > something non-nil by default?  Or maybe modify filter-buffer-substring
> > itself to remove these characters as needed?
> 
> Maybe that can be made to work, but we'd need to add a corresponding
> filter-substring

What for? strings aren't copied by C-w and friends.

> and I'm not sure that we wouldn't get into bugs where we need to
> change the code from buffer-substring to filter-buffer-substring.

Which bugs, and why will we get into them?

Re: bidi-display-reordering is now non-nil by default

[Stefan Monnier]

>> >> So embeddings should not be used too much, and/or should only be used
>> >> dynamically (jit-lock-style) and implicitly (e.g. not copied by
>> >> kill-region).
>> > It depends.  If only part of the embedded text is copied, then the
>> > directional formats should be dropped, indeed.  But if all of it is
>> > copied, they should be kept, to preserve the visual appearance.
>> Sounds difficult to implement reliably.
> What difficulties do you have in mind?

See below.

>> > So I guess we will need to set filter-buffer-substring-functions to
>> > something non-nil by default?  Or maybe modify filter-buffer-substring
>> > itself to remove these characters as needed?
>> Maybe that can be made to work, but we'd need to add a corresponding
>> filter-substring
> What for? strings aren't copied by C-w and friends.

Most substrings are created non-interactively, many of them via
buffer-substring but others via just substring (sometimes/often on the
result of buffer-subtring).

>> and I'm not sure that we wouldn't get into bugs where we need to
>> change the code from buffer-substring to filter-buffer-substring.
> Which bugs, and why will we get into them?

Just think of the "Foo (default bar): " prompts, where `bar' often comes
from some buffer via buffer-substring/match-string.


        Stefan

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Thu, 25 Aug 2011 00:38:51 -0400
> 
> >> Yes, it's a "go ahead" for the "treat any `space' display property as
> >> a segment separator".
> > Well, in light of the above, it sounds like we need to treat such
> > properties as a paragraph separator, to get the effect we need.  I'll
> > take a stab at that.
> 
> Thanks,

Done in revision 105562 on the trunk.  Please give it a try.  Initial
testing with the *Completions* buffer indicates that it does TRT.
I've also removed the calls to bidi-string-mark-left-to-right from
buff-menu.el, as they are no longer needed there.

Re: bidi-display-reordering is now non-nil by default

[Stefan Monnier]

>> +                (put-text-property (point)
>> +				   (progn
>> +				     (insert (bidi-string-mark-left-to-right str))
>> +				     (point))
>> 'mouse-face 'highlight)

The mouse-face property is used to delimit the text, but the above code
ends up marking not just STR but STR+LRM, which is not desirable
(e.g. choose-completion will end up choosing STR+LRM rather than jus STR).


        Stefan

Re: bidi-display-reordering is now non-nil by default

[Štěpán Němec]

On Mon, 22 Aug 2011 21:37:30 +0200
Stefan Monnier wrote:

>>> +                (put-text-property (point)
>>> +				   (progn
>>> +				     (insert (bidi-string-mark-left-to-right str))
>>> +				     (point))
>>> 'mouse-face 'highlight)
>
> The mouse-face property is used to delimit the text, but the above code
> ends up marking not just STR but STR+LRM, which is not desirable
> (e.g. choose-completion will end up choosing STR+LRM rather than jus
> STR).

OTOH, making choose-completion know better than relying on mouse-face
would also have the side-effect of providing more favourable conditions
for fixing bug#8897. ;-)

-- 
Štěpán

Re: bidi-display-reordering is now non-nil by default

[Stefan Monnier]

> OTOH, making choose-completion know better than relying on mouse-face
> would also have the side-effect of providing more favourable conditions
> for fixing bug#8897. ;-)

OTOH bug#8897 is for a use that's pretty far from the normal use case of
the completion code, so I wouldn't make such a significant change just to
accomodate this use case.


        Stefan

Re: bidi-display-reordering is now non-nil by default

[Štěpán Němec]

On Tue, 23 Aug 2011 03:13:23 +0200
Stefan Monnier wrote:

>> OTOH, making choose-completion know better than relying on mouse-face
>> would also have the side-effect of providing more favourable conditions
>> for fixing bug#8897. ;-)
>
> OTOH bug#8897 is for a use that's pretty far from the normal use case of
> the completion code, so I wouldn't make such a significant change just to
> accomodate this use case.

I think I've gathered as much from the fact that the bug is still
unfixed. Which is precisely why I used this precious opportunity to
point out that there might be more reasons. So now there's also the fact
that using the face for this seems like a kludge to Eli (and possibly
others? certainly me included) and gets in the way of other unrelated
code (or at least makes certain "obvious" solutions "non-obvious").

-- 
Štěpán

Re: bidi-display-reordering is now non-nil by default

[Stefan Monnier]

> I think I've gathered as much from the fact that the bug is still
> unfixed. Which is precisely why I used this precious opportunity to
> point out that there might be more reasons. So now there's also the fact
> that using the face for this seems like a kludge to Eli (and possibly
> others? certainly me included) and gets in the way of other unrelated
> code (or at least makes certain "obvious" solutions "non-obvious").

Actually, changing it so that another property is added to delimit the
text wouldn't solve your problem: the mouse-face property would
still also be added and would hence still interfere with your use.


        Stefan

Re: bidi-display-reordering is now non-nil by default

[Štěpán Němec]

On Tue, 23 Aug 2011 20:24:56 +0200
Stefan Monnier wrote:

>> I think I've gathered as much from the fact that the bug is still
>> unfixed. Which is precisely why I used this precious opportunity to
>> point out that there might be more reasons. So now there's also the fact
>> that using the face for this seems like a kludge to Eli (and possibly
>> others? certainly me included) and gets in the way of other unrelated
>> code (or at least makes certain "obvious" solutions "non-obvious").
>
> Actually, changing it so that another property is added to delimit the
> text wouldn't solve your problem: the mouse-face property would
> still also be added and would hence still interfere with your use.

I don't think so. As explained in the bug thread, the problem for me is
that custom text properties (specifically mouse-face in my case) added
to the completion annotations are clobbered. I don't care about the face
of the completion items (although I'd argue they should respect user
additions as well). The argument you raised against not clobbering the
annotations additions and thus allowing face like mouse-face for them
was that the completions code would get confused, because that property
is used to delimit completion items. This argument would disappear if
that was fixed and a separate (non-face) property was used instead.

-- 
Štěpán

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Mon, 22 Aug 2011 15:37:30 -0400
> 
> >> +                (put-text-property (point)
> >> +				   (progn
> >> +				     (insert (bidi-string-mark-left-to-right str))
> >> +				     (point))
> >> 'mouse-face 'highlight)
> 
> The mouse-face property is used to delimit the text, but the above code
> ends up marking not just STR but STR+LRM, which is not desirable
> (e.g. choose-completion will end up choosing STR+LRM rather than jus STR).

I couldn't imagine the mouse-face is used for anything besides the
highlight.  Sounds like a kludge to me.

Anyway, if we will go the bidi-string-mark-left-to-right route, it
would be easy to fix this minor problem by putting the mouse-face on
the string before running it through bidi-string-mark-left-to-right.

Re: bidi-display-reordering is now non-nil by default

[Chong Yidong]

Eli Zaretskii <eliz@gnu.org> writes:

> I don't think so.  The former is a much simpler problem, because it
> has to do with text that normally isn't edited nor submitted to
> external programs.  It is basically a problem of concatenating strings
> subject to certain restrictions wrt their layout on the screen, like
> "make str1 display to the left of str2".
>
> The latter problem is much harder, because it needs a capability to
> reorder only certain portions of a buffer.

Here's an idea: how bout if programming language and related buffers
employ a modified bidi type table in which punctuation characters
(like + " and <) are bidi segment separators (S)?

AFAICT, this hack would fix most of the garbaging problems in source
code as some others like "buffername<1>".

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Chong Yidong <cyd@stupidchicken.com>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, dak@gnu.org,
>         emacs-devel@gnu.org
> Date: Mon, 15 Aug 2011 14:28:23 -0400
> 
> Here's an idea: how bout if programming language and related buffers
> employ a modified bidi type table in which punctuation characters
> (like + " and <) are bidi segment separators (S)?

Currently, the bidi type table is global, so you cannot change it on a
buffer by buffer basis.

More importantly, I don't think it's a good idea to change the bidi
properties in such a radical way (or at all), because we don't have
the resources to investigate all of the consequences.  For example, <
is a mirrored character, so when it's between to R2L characters, it
displays as > and vice versa.  Yes, I know: you'll suggest to change
the mirroring table as well, but how far should we go? do we really
want to create our own UBA?

> AFAICT, this hack would fix most of the garbaging problems in source
> code as some others like "buffername<1>".

But it will do The Wrong Thing if < or > are parts of the buffer name
itself.

Re: bidi-display-reordering is now non-nil by default

[Stefan Monnier]

> More importantly, I don't think it's a good idea to change the bidi
> properties in such a radical way (or at all), because we don't have
> the resources to investigate all of the consequences.  For example, <
> is a mirrored character, so when it's between to R2L characters, it
> displays as > and vice versa.  Yes, I know: you'll suggest to change
> the mirroring table as well, but how far should we go? do we really
> want to create our own UBA?

Furthermore, this buffer-local table should only be used for actual
code, but the default global table should probably be used within
comments and strings.


        Stefan

Re: bidi-display-reordering is now non-nil by default

[Chong Yidong]

Eli Zaretskii <eliz@gnu.org> writes:

> More importantly, I don't think it's a good idea to change the bidi
> properties in such a radical way (or at all), because we don't have
> the resources to investigate all of the consequences.  For example, <
> is a mirrored character, so when it's between to R2L characters, it
> displays as > and vice versa.  Yes, I know: you'll suggest to change
> the mirroring table as well, but how far should we go? do we really
> want to create our own UBA?

The specs allow for a "higher level protocol" to segment text according
to what makes sense for (say) Lisp syntax, so anything that displays
most realistic Lisp code correctly would be good enough.

I guess this is not realistic to implement for 24.1, though, so since
there is no near-term solution, how bout turning off bidi display
reordering for prog-mode buffers?

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Chong Yidong <cyd@stupidchicken.com>
> Cc: monnier@iro.umontreal.ca, dak@gnu.org, emacs-devel@gnu.org
> Date: Mon, 15 Aug 2011 22:02:53 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > More importantly, I don't think it's a good idea to change the bidi
> > properties in such a radical way (or at all), because we don't have
> > the resources to investigate all of the consequences.  For example, <
> > is a mirrored character, so when it's between to R2L characters, it
> > displays as > and vice versa.  Yes, I know: you'll suggest to change
> > the mirroring table as well, but how far should we go? do we really
> > want to create our own UBA?
> 
> The specs allow for a "higher level protocol" to segment text according
> to what makes sense for (say) Lisp syntax, so anything that displays
> most realistic Lisp code correctly would be good enough.

"Higher level protocols" don't include futzing with bidirectional
properties of characters.  "Higher level protocol" means some means to
determine segment boundaries other than segment separator characters
that are part of the text.  Translated into Emacs-speak, this means we
need a variable that, when bound to some special value, instructs the
reordering engine to treat certain characters as segment separators.

> I guess this is not realistic to implement for 24.1

If we go with a variable described above, it's actually a very simple
job, so just say the word.  However, I suggest that before doing this
we come up with a more detailed plan: suppose you have this variable,
how would you solve the problems discussed here with its help?  The
devil is in the details.

> since there is no near-term solution, how bout turning off bidi
> display reordering for prog-mode buffers?

What for?  No one complained about it yet, and leaving it on helps
find bugs and inefficiencies in the bidi display engine.  So my vote
is NAY.

Re: bidi-display-reordering is now non-nil by default

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Chong Yidong <cyd@stupidchicken.com>
>
>> since there is no near-term solution, how bout turning off bidi
>> display reordering for prog-mode buffers?
>
> What for?  No one complained about it yet, and leaving it on helps
> find bugs and inefficiencies in the bidi display engine.  So my vote
> is NAY.

Sounds like the plan is to make Emacs painful to use for everyone in
order to make it a priority for L2R users to solve R2L problems.

That may be a pretest strategy.  But I don't think it is a good idea to
stay with it for the betas, or even the release.

-- 
David Kastrup

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Tue, 16 Aug 2011 09:07:16 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Chong Yidong <cyd@stupidchicken.com>
> >
> >> since there is no near-term solution, how bout turning off bidi
> >> display reordering for prog-mode buffers?
> >
> > What for?  No one complained about it yet, and leaving it on helps
> > find bugs and inefficiencies in the bidi display engine.  So my vote
> > is NAY.
> 
> Sounds like the plan is to make Emacs painful to use for everyone in
> order to make it a priority for L2R users to solve R2L problems.

If L2R users don't want bidirectional support any place near them, to
the degree that they are unwilling to help make it better at the cost
of some moderate effort, please tell so now.  I will then rip out
everything I did, and will forbid the FSF to ever use that code in any
version of Emacs they distribute.  That won't make up for the 2 years
of efforts I invested in making this happen, but at least I will cut
my losses and won't have to go through the ordeal of arguing with one
David Kastrup anymore.

Re: bidi-display-reordering is now non-nil by default

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>> Date: Tue, 16 Aug 2011 09:07:16 +0200
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> From: Chong Yidong <cyd@stupidchicken.com>
>> >
>> >> since there is no near-term solution, how bout turning off bidi
>> >> display reordering for prog-mode buffers?
>> >
>> > What for?  No one complained about it yet, and leaving it on helps
>> > find bugs and inefficiencies in the bidi display engine.  So my vote
>> > is NAY.
>> 
>> Sounds like the plan is to make Emacs painful to use for everyone in
>> order to make it a priority for L2R users to solve R2L problems.
>
> If L2R users don't want bidirectional support any place near them,

You are putting words into my mouth.

> to the degree that they are unwilling to help make it better at the
> cost of some moderate effort, please tell so now.

Most users are _unable_ to help.  The pretest versions are for mostly
intended for those who _are_ likely able to help.  The betas and the
final releases, however, are also for everybody else.

We have refrained from enabling features like font-locking by default in
releases until the costs of using them for everyone were under control.

Nobody is arguing to remove R2L support.  The question is about what
defaults are sensible to put into a release.  You have yourself worked
rather hard to make sure that one can have performance comparable to
before by disabling bidi support altogether.

At the current point of time, I don't think that anybody suggests
bidi-display-reordering be set to nil in the release version: work is
good enough that this need not be an issue.

Your standard reaction for regressions encountered in connection with
discussions about both performance problems as well as more fine-grained
defaults is threats.  That is not productive.  I understand that you
have invested a large amount of work into R2L typesetting, and I
understand that it is vital for your personal resources (including your
motivation) that others join in the effort.

> I will then rip out everything I did, and will forbid the FSF to ever
> use that code in any version of Emacs they distribute.  That won't
> make up for the 2 years of efforts I invested in making this happen,
> but at least I will cut my losses and won't have to go through the
> ordeal of arguing with one David Kastrup anymore.

Which is silly since I am not even a developer of Emacs anymore.  I
handed back my commit rights years ago after a similar discussion with
you.

I am now unsubscribing from the Emacs developer list in order to save
you having to worry about my person.

But you really should think whether your way of dealing with technical
problems by ostracizing is doing yourself a favor, let alone anybody
else.

-- 
David Kastrup

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Tue, 16 Aug 2011 12:01:22 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: David Kastrup <dak@gnu.org>
> >> Date: Tue, 16 Aug 2011 09:07:16 +0200
> >> 
> >> Eli Zaretskii <eliz@gnu.org> writes:
> >> 
> >> >> From: Chong Yidong <cyd@stupidchicken.com>
> >> >
> >> >> since there is no near-term solution, how bout turning off bidi
> >> >> display reordering for prog-mode buffers?
> >> >
> >> > What for?  No one complained about it yet, and leaving it on helps
> >> > find bugs and inefficiencies in the bidi display engine.  So my vote
> >> > is NAY.
> >> 
> >> Sounds like the plan is to make Emacs painful to use for everyone in
> >> order to make it a priority for L2R users to solve R2L problems.
> >
> > If L2R users don't want bidirectional support any place near them,
> 
> You are putting words into my mouth.

No, I'm reading what they say loud and clear.

> > to the degree that they are unwilling to help make it better at the
> > cost of some moderate effort, please tell so now.
> 
> Most users are _unable_ to help.

Wrong.  They help _a_lot_ by using the display features and reporting
bugs.  I'm grateful to all of them, because I could never find these
problems myself.

> We have refrained from enabling features like font-locking by default in
> releases until the costs of using them for everyone were under control.

Whether or not the bidi display is good enough to hit the street
should be an issue near the end of the pretest, which has not even
started yet.  So raising these claims now is unjustified and premature
at best.

> Nobody is arguing to remove R2L support.  The question is about what
> defaults are sensible to put into a release.  You have yourself worked
> rather hard to make sure that one can have performance comparable to
> before by disabling bidi support altogether.

The possibility to disable bidi support is a debugging aid, that's
all.  That is why it is not a user variable.  Don't read anything else
into that.

> Your standard reaction for regressions encountered in connection with
> discussions about both performance problems as well as more fine-grained
> defaults is threats.

Hogwash.  My standard reaction to that is fixing every regression that
is being reported.  Anyone who looks at the bug tracker will see that.
When I disagree with suggestions to change the defaults, I explain
why.  I would expect my opinions and explanations to have some
considerable weight, precisely _because_ I'm working on this alone,
and therefore know much more about these issues than anyone else here.

> I understand that you have invested a large amount of work into R2L
> typesetting, and I understand that it is vital for your personal
> resources (including your motivation) that others join in the
> effort.

I have no problems keeping doing this alone.  But I need to know that
I have the support of the Emacs community in doing so.  It is okay to
ask questions about the technical solutions that I implement or
suggest, or suggest alternatives.  But as soon as my _motives_ are
questioned, as soon as someone dares to suggest that I'm deliberately
putting users through suffering for some personal reasons, that is the
sign of lack of support.  I will not go on without support.  I refuse
to work on features that will not be used in a released Emacs, for no
good reason except that it brings more complexity and perhaps some
moderate slowdown.

If someone thinks that complexity and some slowdown can be avoided
while doing so much more, then they are dreaming.  Every significant
feature we put in Emacs always causes similar effects.  Testing them
in a variety of environments and workflows is the only way of getting
them improved and enhanced for the users.  This feature is no
different.

> But you really should think whether your way of dealing with technical
> problems by ostracizing is doing yourself a favor, let alone anybody
> else.

I have no problem with your technical comments, never had them.  What
triggered this exchange was a nasty personal attack on myself and my
motives, which had nothing to do with technicalities.  That I will not
tolerate, not from you, not from anyone else.

Re: bidi-display-reordering is now non-nil by default

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

> Translated into Emacs-speak, this means we need a variable that, when
> bound to some special value, instructs the reordering engine to treat
> certain characters as segment separators.

They already exist: comment-start and comment-end, for example.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: bidi-display-reordering is now non-nil by default

[David Kastrup]

Andreas Schwab <schwab@linux-m68k.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> Translated into Emacs-speak, this means we need a variable that, when
>> bound to some special value, instructs the reordering engine to treat
>> certain characters as segment separators.
>
> They already exist: comment-start and comment-end, for example.

I am having a somewhat hard time imagining it as a good thing if Emacs
displays source code that does no longer correspond to a reasonably
straightforward manner of printing the file.  reordering source into
something that "makes more sense" seems quite more invasive than mere
font-locking.

I am not fond of the idea of writing code that gets unreadable unless
you read it back with Emacs, or even a specific version of Emacs or
specific settings.

It may partly be the problem space of the language specification:
something like HTML should likely give ways of using of directional
marks as source formatting aids without affecting the content payload.

-- 
David Kastrup

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Tue, 16 Aug 2011 09:54:18 +0200
> 
> Andreas Schwab <schwab@linux-m68k.org> writes:
> 
> > Eli Zaretskii <eliz@gnu.org> writes:
> >
> >> Translated into Emacs-speak, this means we need a variable that, when
> >> bound to some special value, instructs the reordering engine to treat
> >> certain characters as segment separators.
> >
> > They already exist: comment-start and comment-end, for example.
> 
> I am having a somewhat hard time imagining it as a good thing if Emacs
> displays source code that does no longer correspond to a reasonably
> straightforward manner of printing the file.  reordering source into
> something that "makes more sense" seems quite more invasive than mere
> font-locking.

Please explain what you mean by "no longer corresponds".  If a comment
or a string in a source file use R2L scripts, how would you like Emacs
to display them?  If the answer is in their original logical order,
then how is the user staring at the source supposed to read those
comments and strings?

As for printing, don't modern printers and associated software already
reorder text when they print?  If not, we may be able to use Emacs
facilities to DTRT here.

> I am not fond of the idea of writing code that gets unreadable unless
> you read it back with Emacs, or even a specific version of Emacs or
> specific settings.

??? The discussion was about _displaying_, no one suggested writing
anything to the file.  The idea was to use _existing_ characters or
strings that delimit the relevant portions of source as a cue for
reordering those delimited parts in some way that makes them readable.

Re: bidi-display-reordering is now non-nil by default

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>> Date: Tue, 16 Aug 2011 09:54:18 +0200
>> 
>> Andreas Schwab <schwab@linux-m68k.org> writes:
>> 
>> > Eli Zaretskii <eliz@gnu.org> writes:
>> >
>> >> Translated into Emacs-speak, this means we need a variable that, when
>> >> bound to some special value, instructs the reordering engine to treat
>> >> certain characters as segment separators.
>> >
>> > They already exist: comment-start and comment-end, for example.
>> 
>> I am having a somewhat hard time imagining it as a good thing if Emacs
>> displays source code that does no longer correspond to a reasonably
>> straightforward manner of printing the file.  reordering source into
>> something that "makes more sense" seems quite more invasive than mere
>> font-locking.
>
> Please explain what you mean by "no longer corresponds".  If a comment
> or a string in a source file use R2L scripts, how would you like Emacs
> to display them?

With the normal bidi algorithm.

> As for printing, don't modern printers and associated software already
> reorder text when they print?

According to the normal bidi algorithm, but hardly by recognizing and
treating comment fields on their own.

> If not, we may be able to use Emacs facilities to DTRT here.

If "the right thing" is different from the rest of the world, working
with the text is tied into Emacs.

>> I am not fond of the idea of writing code that gets unreadable unless
>> you read it back with Emacs, or even a specific version of Emacs or
>> specific settings.
>
> ??? The discussion was about _displaying_, no one suggested writing
> anything to the file.  The idea was to use _existing_ characters or
> strings that delimit the relevant portions of source as a cue for
> reordering those delimited parts in some way that makes them readable.

If the byte stream in the file represents an unreadable result, then
Emacs should better facilitate creating a byte stream representing a
readable result (for example, by semiautomatically inserting direction
marks when the result would otherwise start looking bad) instead of
presenting a more readable result that does not corresponding to the
file contents in a straightforward manner.

When used as a mere file viewer, there is something to be said for using
higher level information than the bidi algorithm in order to give a
rendition that is better than the original.

But if Emacs is used as an editor, a writing tool, it should help to
write readable source, and presenting a more readable letter order than
the bidi algorithm or a printer or a browser could come up with is
misleading the writer.

-- 
David Kastrup

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Tue, 16 Aug 2011 11:40:28 +0200
> 
> If the byte stream in the file represents an unreadable result, then
> Emacs should better facilitate creating a byte stream representing a
> readable result (for example, by semiautomatically inserting direction
> marks when the result would otherwise start looking bad) instead of
> presenting a more readable result that does not corresponding to the
> file contents in a straightforward manner.

That's fine with me, but I thought people were saying that compilers
and interpreters might not like these directional controls.  If that's
not true, using them would be the most natural and simple solution,
IMO.

Does someone know how other programming IDEs treat R2L text in
comments in strings?

> But if Emacs is used as an editor, a writing tool, it should help to
> write readable source, and presenting a more readable letter order than
> the bidi algorithm or a printer or a browser could come up with is
> misleading the writer.

Assuming that other programming editors don't have similar features
that don't depend on directional controls, I agree.  Displaying text
in a way that is very different from other similar tools is not a good
idea.

Re: bidi-display-reordering is now non-nil by default

[Stefan Monnier]

> Does someone know how other programming IDEs treat R2L text in
> comments in strings?

Indeed, I think we first need to be more clear about the problem with
which we're concerned.
AFAIK there are a few different cases:
- R2L in comments, strings, and doc.
- R2L in identifiers.
If we manage to mark comment/string/doc delimiters as
bidi-field-delimiters, then the first problem is solved.
But the second is still open: I wouldn't want

  (foo ARGA ARGB)

to be displayed as

  (foo (BGRA AGRA


        Stefan

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: Chong Yidong <cyd@stupidchicken.com>,  dak@gnu.org,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Tue, 16 Aug 2011 09:40:45 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Translated into Emacs-speak, this means we need a variable that, when
> > bound to some special value, instructs the reordering engine to treat
> > certain characters as segment separators.
> 
> They already exist: comment-start and comment-end, for example.

These are strings, not characters.  I meant single characters.
Making that a string would make the job of implementing such a
high-level protocol much harder, because the reordering engine always
considers only one character, with a very few exceptions.

Also, a segment cannot span more than a single line (up to a newline),
so multi-line comments cannot use this solution.

What I had in mind was not comments and strings in program source, but
issues with display of foobar<1> buffer names and lines in
mail-summary buffers, which I thought was what Chong had in mind.  It
could also be a solution for displaying XML, although that use case
needs more thorough analysis, to see if this would really be a good
solution.

The key issue here is whether those same characters can appear in
context where we do want them to be reordered as "normal" punctuation.
If the answer is YES, then this idea won't hold water.

Re: bidi-display-reordering is now non-nil by default

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andreas Schwab <schwab@linux-m68k.org>
>> Cc: Chong Yidong <cyd@stupidchicken.com>,  dak@gnu.org,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
>> Date: Tue, 16 Aug 2011 09:40:45 +0200
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > Translated into Emacs-speak, this means we need a variable that, when
>> > bound to some special value, instructs the reordering engine to treat
>> > certain characters as segment separators.
>> 
>> They already exist: comment-start and comment-end, for example.
>
> These are strings, not characters.  I meant single characters.

You can't have that.  The field separators are multi-character strings.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: cyd@stupidchicken.com,  dak@gnu.org,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Tue, 16 Aug 2011 11:10:33 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Andreas Schwab <schwab@linux-m68k.org>
> >> Cc: Chong Yidong <cyd@stupidchicken.com>,  dak@gnu.org,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> >> Date: Tue, 16 Aug 2011 09:40:45 +0200
> >> 
> >> Eli Zaretskii <eliz@gnu.org> writes:
> >> 
> >> > Translated into Emacs-speak, this means we need a variable that, when
> >> > bound to some special value, instructs the reordering engine to treat
> >> > certain characters as segment separators.
> >> 
> >> They already exist: comment-start and comment-end, for example.
> >
> > These are strings, not characters.  I meant single characters.
> 
> You can't have that.  The field separators are multi-character strings.

If all of them can be made to behave like segment separators, then
Chong's idea will still work.  If not, then it won't.

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: Chong Yidong <cyd@stupidchicken.com>,  dak@gnu.org,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Tue, 16 Aug 2011 09:40:45 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Translated into Emacs-speak, this means we need a variable that, when
> > bound to some special value, instructs the reordering engine to treat
> > certain characters as segment separators.
> 
> They already exist: comment-start and comment-end, for example.

These are strings, not characters.  I meant single characters.
Making that a string would make the job of implementing such a
high-level protocol much harder, because the reordering engine always
considers only one character, with a very few exceptions.

Also, a segment cannot span more than a single line (up to a newline),
so multi-line comments cannot use this solution.

What I had in mind was not comments and strings in program source, but
issues with display of foobar<1> buffer names and lines in
mail-summary buffers, which I thought was what Chong had in mind.  It
could also be a solution for displaying XML, although that use case
needs more thorough analysis, to see if this would really be a good
solution.

The key issue here is whether those same characters can appear in
context where we do want them to be reordered as "normal" punctuation.
If the answer is YES, then this idea won't hold water.

Re: bidi-display-reordering is now non-nil by default

[Stefan Monnier]

> "Higher level protocols" don't include futzing with bidirectional
> properties of characters.  "Higher level protocol" means some means to
> determine segment boundaries other than segment separator characters
> that are part of the text.  Translated into Emacs-speak, this means we
> need a variable that, when bound to some special value, instructs the
> reordering engine to treat certain characters as segment separators.

Maybe we then need a way to declare some chars (such as SPC,
punctuation, string markers, comment markers, ...) as
segment boundaries.

>> since there is no near-term solution, how bout turning off bidi
>> display reordering for prog-mode buffers?
> What for?  No one complained about it yet, and leaving it on helps
> find bugs and inefficiencies in the bidi display engine.  So my vote
> is NAY.

Agreed.


        Stefan

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Chong Yidong <cyd@stupidchicken.com>,  dak@gnu.org,  emacs-devel@gnu.org
> Date: Tue, 16 Aug 2011 10:03:15 -0400
> 
> > "Higher level protocols" don't include futzing with bidirectional
> > properties of characters.  "Higher level protocol" means some means to
> > determine segment boundaries other than segment separator characters
> > that are part of the text.  Translated into Emacs-speak, this means we
> > need a variable that, when bound to some special value, instructs the
> > reordering engine to treat certain characters as segment separators.
> 
> Maybe we then need a way to declare some chars (such as SPC,
> punctuation, string markers, comment markers, ...) as
> segment boundaries.

As I wrote, implementing this for an arbitrary set of characters is
easy.  But the problem is that any character which we declare a
segment separator will behave like that everywhere in that buffer.  If
the same characters appear in a different context, where we do want
the reordering, we are screwed.

For example, suppose we declare SPC a segment separator.  Then this:

  (foo ARGA ARGB)

will be displayed correctly, but blanks in doc strings in the same
buffer will mess up the display, because something like this:

  "FOO BAR BAZ"

will be displayed like this:

  "OOF RAB ZAB"

whereas we want this:

  "ZAB RAB OOF"

Re: bidi-display-reordering is now non-nil by default

[Chong Yidong]

Eli Zaretskii <eliz@gnu.org> writes:

>> since there is no near-term solution, how bout turning off bidi
>> display reordering for prog-mode buffers?
>
> What for?  No one complained about it yet

I think I mentioned in an earlier message that the presence of RTL
characters mucks up the display of Emacs Lisp buffers.  This happend to
me while writing code containing RTL strings to test
string-mark-left-to-right.

> leaving it on helps find bugs and inefficiencies in the bidi display
> engine.

I understand your point, but it's not realistic to expect anyone to
"find bugs and inefficiencies" in source code buffers right now.  Those
buffers aren't displayed in an intelligible manner, because they require
missing "higher-level protocols" to display properly.

It makes a lot of sense for Emacs 24.1 to provide just the "bare" UAX9
algorithm, enabled by default in text modes (where bare UAX9 will likely
DTRT) and special modes (where Emacs can segment by hand if necessary);
with the higher-level segmentation functionality postphoned to the next
release.  In prog modes, which require that higher-level segmentation
functionality, it then makes sense to disable bidi display for now.

This would also give a lot more time to study different ways of
implementing segmentation (e.g. definining segmentation characters vs
extending the bidi code to recognize text properties).

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Chong Yidong <cyd@stupidchicken.com>
> Cc: monnier@iro.umontreal.ca, dak@gnu.org, emacs-devel@gnu.org
> Date: Tue, 16 Aug 2011 11:48:54 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> since there is no near-term solution, how bout turning off bidi
> >> display reordering for prog-mode buffers?
> >
> > What for?  No one complained about it yet
> 
> I think I mentioned in an earlier message that the presence of RTL
> characters mucks up the display of Emacs Lisp buffers.  This happend to
> me while writing code containing RTL strings to test
> string-mark-left-to-right.

Won't a newline and/or LRM/RLM (the latter inside the string) fix
that?  If not, please show me the relevant examples.

In general, all reordering information is tossed at every newline and
restarted anew for the next line.  So judicious placement of newlines
should do the trick in most cases.

> Those buffers aren't displayed in an intelligible manner, because
> they require missing "higher-level protocols" to display properly.

I think you exaggerate.  Both strings and comments are mostly
displayed correctly, and a few nasty surprises can be fixed by
inserting newlines and sometimes directional control characters.
Sure, there's place for improvement: Emacs should not force the user
to format the source in some specific manner, for it to display
correctly.  But for someone who actually uses R2L characters in
comments and strings, this is a definite improvement compared to Emacs
23, where they had to read them backwards.

> In prog modes, which require that higher-level segmentation
> functionality, it then makes sense to disable bidi display for now.

I disagree (see above), but I'll go with anything you and Stefan
decide.

> This would also give a lot more time to study different ways of
> implementing segmentation (e.g. definining segmentation characters vs
> extending the bidi code to recognize text properties).

I don't think we should feel pressed to resolve this before Emacs
24.1, since the problem is not as acute as it may seem at first
glance.  So we have that time anyway.

Re: bidi-display-reordering is now non-nil by default

[Chong Yidong]

Eli Zaretskii <eliz@gnu.org> writes:

> Won't a newline and/or LRM/RLM (the latter inside the string) fix
> that?  In general, all reordering information is tossed at every
> newline and restarted anew for the next line.  So judicious placement
> of newlines should do the trick in most cases.

Having to juggle newlines when editing source code is no good.

The ideal long-term solution, I think, is to make the bidi display code
aware of text properties, as several people have already suggested.
Then it should be easy to exploit font-lock to give reasonably correct
bidi segmentation, e.g. by treating font-lock-comment-face and
font-lock-string-face boundaries as bidi segmentation boundaries.

For now, I don't feel strongly about the idea of turning off bidi
display in prog modes.  But if we don't, we should at least document
some of the pitfalls discussed in this thread, and maybe state that
having RTL script in source code buffers is currently a bad idea.

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Chong Yidong <cyd@stupidchicken.com>
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> Date: Tue, 16 Aug 2011 18:24:24 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Won't a newline and/or LRM/RLM (the latter inside the string) fix
> > that?  In general, all reordering information is tossed at every
> > newline and restarted anew for the next line.  So judicious placement
> > of newlines should do the trick in most cases.
> 
> Having to juggle newlines when editing source code is no good.

It's an annoyance, I agree.  We should lift this limitation as soon as
we can.

> The ideal long-term solution, I think, is to make the bidi display code
> aware of text properties, as several people have already suggested.

I'm afraid making the reordering engine aware of all text properties
will considerably slow down redisplay, due to the need to check
character properties very frequently.  It also runs a high risk of
completely blending the reordering code with the display engine, which
will make them both very hard to maintain; currently, they are clearly
separated.

In addition, the current design of bidi.c does not support starting
and stopping reordering on arbitrary character positions; it only does
that at BEGV and ZV.  If possible, I'd like to avoid changing that.

The idea I currently favor is to use the `display' properties for this
purpose.  bidi.c already takes notice of those `display' properties
which replace the underlying text with something else, like an image
or another text.  The text covered by such properties is reordered as
a single long character, and if a property is a string, it is then
reordered separately by pushing the iterator object onto a stack and
restarting the iteration, including reordering, anew for the display
string.

So suppose we define a new display spec, something like

 'display '(bidi-reorder BASE-DIRECTION)

where BASE-DIRECTION, either left-to-right or right-to-left, is the
"paragraph direction" to be in effect while reordering the text
covered by the display spec.  Then we add code to the display engine
to implement this spec by pushing the iterator, reinitializing the
iteration for the span of this display property, then popping it when
it's done.  In effect, we create a "display string" out of a chunk of
buffer text, and then reorder it separately.

With this feature, a major mode can put such display properties on
whichever parts of buffer text it wants.  This will work regardless of
whether the buffer itself has bidi-display-reordering set to nil or t,
because the implementation of such display spec will force reordering
for the covered text regardless of the parent buffer.

> Then it should be easy to exploit font-lock to give reasonably correct
> bidi segmentation, e.g. by treating font-lock-comment-face and
> font-lock-string-face boundaries as bidi segmentation boundaries.

We should be very careful with reusing font-lock as basis for
reordering, because the user has too much knobs to control font-lock.
For example, few of the font-lock features speed up redisplay by
deferring fontification to a later time.  With font-lock, this just
displays text in the default face; with reordering, it will flush
incorrectly rendered text for a perceptible amount of time.  I'm not
sure it's a good idea.

There are also people who disable font-lock completely.

We could use font-lock faces, when they exist, as a cue for placing
the `display' properties described above, though.

> For now, I don't feel strongly about the idea of turning off bidi
> display in prog modes.  But if we don't, we should at least document
> some of the pitfalls discussed in this thread, and maybe state that
> having RTL script in source code buffers is currently a bad idea.

I'll add this to my todo and revisit that during the pretest.

Re: bidi-display-reordering is now non-nil by default

[Juri Linkov]

>  'display '(bidi-reorder BASE-DIRECTION)

I wonder is it possible to add a new property `dir', so you have
to check just two properties `display' and `dir' in bidi.c?

In HTML, the `dir' attribute specifies the base direction (ltr, rtl) of text:

  <p dir="ltr">text</p>
  <p dir="rtl">text</p>

Info: http://www.w3.org/International/tutorials/bidi-xhtml/

In Emacs it would be good to do the same with similar text properties:

  (propertize "text" 'dir 'ltr)
  (propertize "text" 'dir 'rtl)

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Juri Linkov <juri@jurta.org>
> Cc: Chong Yidong <cyd@stupidchicken.com>,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Wed, 17 Aug 2011 12:34:32 +0300
> 
> >  'display '(bidi-reorder BASE-DIRECTION)
> 
> I wonder is it possible to add a new property `dir', so you have
> to check just two properties `display' and `dir' in bidi.c?
> 
> In HTML, the `dir' attribute specifies the base direction (ltr, rtl) of text:
> 
>   <p dir="ltr">text</p>
>   <p dir="rtl">text</p>

Your `dir' is my BASE-DIRECTION.  That is, modes that support HTML
will put a display property with the above form of spec on "text", and
set the BASE-DIRECTION value according to dir=.  So my proposal should
be sufficient for rendering HTML and XML as well.

Re: bidi-display-reordering is now non-nil by default

[Chong Yidong]

Eli Zaretskii <eliz@gnu.org> writes:

> I'm afraid making the reordering engine aware of all text properties
> will considerably slow down redisplay, due to the need to check
> character properties very frequently.  It also runs a high risk of
> completely blending the reordering code with the display engine, which
> will make them both very hard to maintain; currently, they are clearly
> separated.

No, the lookup would be done at the redisplay engine level, not the
reordering engine level: add a new entry in it_props[] for handling a
(say) `bidi-override' text property.  Emacs would process this during
the step in redisplay where it handles other properties (like faces and
invisibility), and record the information into the iterator.  The bidi
code would take it from there.

That should not be particularly performance intensive; we are already
doing an equal or greater amount of work handling things like face
properties.

>> Then it should be easy to exploit font-lock to give reasonably correct
>> bidi segmentation, e.g. by treating font-lock-comment-face and
>> font-lock-string-face boundaries as bidi segmentation boundaries.
>
> We should be very careful with reusing font-lock as basis for
> reordering, because the user has too much knobs to control font-lock.
> For example, few of the font-lock features speed up redisplay by
> deferring fontification to a later time.  With font-lock, this just
> displays text in the default face; with reordering, it will flush
> incorrectly rendered text for a perceptible amount of time.  I'm not
> sure it's a good idea.

The fundamental issue is that correctly segmenting source code requires
knowledge of the underlying syntax.  Sure, it's possible to come up with
some hacks that "mostly" work, but font lock is already there, so we
ought to try to use it first.

For this reason, I'm not about concerned about the deferred
fontification issue: if you want Emacs to segment properly, you'd want
it to do an amount of work equivalent to font-lock anyway.

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Chong Yidong <cyd@stupidchicken.com>
> Date: Wed, 17 Aug 2011 18:32:46 -0400
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I'm afraid making the reordering engine aware of all text properties
> > will considerably slow down redisplay, due to the need to check
> > character properties very frequently.  It also runs a high risk of
> > completely blending the reordering code with the display engine, which
> > will make them both very hard to maintain; currently, they are clearly
> > separated.
> 
> No, the lookup would be done at the redisplay engine level, not the
> reordering engine level: add a new entry in it_props[] for handling a
> (say) `bidi-override' text property.  Emacs would process this during
> the step in redisplay where it handles other properties (like faces and
> invisibility), and record the information into the iterator.  The bidi
> code would take it from there.

This won't work, not with the way the reordering engine is currently
integrated with redisplay.  The reason is that above the reordering
level, the iteration through buffer text is non-linear.  Your
suggestion assumes that the redisplay iterator will bump into this new
text property _before_ it processes the text which follows it.  But
this assumption is false because of the non-linear scanning of the
buffer text.

Let me show an example to illustrate how the bidirectional display
handles text properties.  Suppose you have the following buffer text
(as usual, capital letters mean R2L characters):

   000000111110000
   abcde ABCDE xyz
         ^    ^

The number above each character shows the text properties of the
characters; 0 means no properties, 1 means some specific property.
This example shows only one property, spanning only the R2L
characters; the real-life examples can be much more complex.  The '^'
characters below show the "stop positions" computed by the iterator --
those are the buffer positions where display engine should process the
text property by calling one or more handlers in the it_props[] array,
filling the iterator with attributes necessary for displaying the text
until the next "stop position".

To move from the blank character between `e' and `A' to the next
character in visual order, the display iterator calls the reordering
engine.  When it does that, the first (leftmost) "stop position" was
not yet acted upon, because the current iterator position is smaller
than that stop.  When the call to the reordering engine returns, it
sets the iterator position at `E', since the ABCDE part should be
displayed as EDCBA on the screen.  Oops! we just missed the "stop
position".  What happens next is the redisplay engine realizes that
the stop position was missed, so it scans back to find the last "stop
position" preceding `E' (since there could be other text properties or
overlays in-between), and then handles it using the handlers in
it_props[]; see handle_stop_backwards for how this is done.  Then it
can deliver `E' with the right attributes, and continue delivering all
the successive characters, until it crosses some "stop position"
again, either going forward or backward.

This is why it won't work to control reordering with text properties:
by the time the redisplay engine realizes that there's another text
property to apply, a crucial part of reordering has already happened.
The bidi_it structure that is part of the iterator already has all the
information about reordering of "ABCDE", having scanned it all inside
a single call to bidi_move_to_visually_next.  That scan entirely
ignores all text properties except one: the `display' property, and
then only if its value will cause the covered text to be replaced by
something else, like an image or a string.

It would be possible, of course, to have the handler of the
`bidi-override' property to toss all the reordering information,
reposition to before `A' and start anew.  But that's a terrible waste
of cycles, especially if the text covered by that property is not so
short.  The waste is not only in that we will have to throw away
information we already gathered at some cost, but also because
repositioning the iterator to an arbitrary place means we need to
restart the bidi iteration from the beginning of the line in order to
have the correct state of the bidi iterator needed to continue from
that place; see get_visually_first_element for the details.

> >> Then it should be easy to exploit font-lock to give reasonably correct
> >> bidi segmentation, e.g. by treating font-lock-comment-face and
> >> font-lock-string-face boundaries as bidi segmentation boundaries.
> >
> > We should be very careful with reusing font-lock as basis for
> > reordering, because the user has too much knobs to control font-lock.
> > For example, few of the font-lock features speed up redisplay by
> > deferring fontification to a later time.  With font-lock, this just
> > displays text in the default face; with reordering, it will flush
> > incorrectly rendered text for a perceptible amount of time.  I'm not
> > sure it's a good idea.
> 
> The fundamental issue is that correctly segmenting source code requires
> knowledge of the underlying syntax.  Sure, it's possible to come up with
> some hacks that "mostly" work, but font lock is already there, so we
> ought to try to use it first.

Font-lock just uses regexps and syntax tables.  Everything else in
font-lock is meant to avoid the annoyingly long delay it takes to
fully fontify a large buffer.  What I'm saying is that, apart from
using regexps and syntax tables, the considerations and trade-offs
that are valid for font-lock are not necessarily valid for
bidirectional display.

> For this reason, I'm not about concerned about the deferred
> fontification issue: if you want Emacs to segment properly, you'd want
> it to do an amount of work equivalent to font-lock anyway.

Amount of work is the least of my concerns in this regard.  I'm
worried about the effect the temporarily incorrect display will have
on users.

Re: bidi-display-reordering is now non-nil by default

[Chong Yidong]

Eli Zaretskii <eliz@gnu.org> writes:

> What happens next is the redisplay engine realizes that the stop
> position was missed, so it scans back to find the last "stop position"
> preceding `E' (since there could be other text properties or overlays
> in-between), and then handles it using the handlers in it_props[]; see
> handle_stop_backwards for how this is done.  Then it can deliver `E'
> with the right attributes, and continue delivering all the successive
> characters, until it crosses some "stop position" again, either going
> forward or backward.

Thanks for the explanation; I think I understand the problem.  But if
this is a fundamental limitation, that would be extremely disappointing.
The use of HTML spans for bidi segmentation is (IIUC) fairly well-known,
and character properties are the most straightforward analog in Emacs.

Here's another rough idea: when handle_stop is computing the next stop
position, it could call another function (analogous to compute_stop_pos)
to find and record the boundary of the next bidi-segmenting character
property---i.e. a "bidi stop position".  Then the reordering function
could look at this to help figure out the segmentation breaks, thus
avoiding advancing past this position.  Would something like that work?

> Amount of work is the least of my concerns in this regard.  I'm
> worried about the effect the temporarily incorrect display will have
> on users.

I don't recall the last time I ran into a problem with deferred
font-lock.  I seriously doubt this is a real issue.

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Chong Yidong <cyd@stupidchicken.com>
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> Date: Thu, 18 Aug 2011 13:13:40 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > What happens next is the redisplay engine realizes that the stop
> > position was missed, so it scans back to find the last "stop position"
> > preceding `E' (since there could be other text properties or overlays
> > in-between), and then handles it using the handlers in it_props[]; see
> > handle_stop_backwards for how this is done.  Then it can deliver `E'
> > with the right attributes, and continue delivering all the successive
> > characters, until it crosses some "stop position" again, either going
> > forward or backward.
> 
> Thanks for the explanation; I think I understand the problem.  But if
> this is a fundamental limitation, that would be extremely disappointing.
> The use of HTML spans for bidi segmentation is (IIUC) fairly well-known,
> and character properties are the most straightforward analog in Emacs.

What's wrong with using the `display' property with a special value,
as I suggested a few mails ago?  It _is_ a character property, isn't
it?

> Here's another rough idea: when handle_stop is computing the next stop
> position, it could call another function (analogous to compute_stop_pos)
> to find and record the boundary of the next bidi-segmenting character
> property---i.e. a "bidi stop position".  Then the reordering function
> could look at this to help figure out the segmentation breaks, thus
> avoiding advancing past this position.  Would something like that work?

Yes.  But using the `display' property means fewer changes, because we
already look for it during bidi iteration, and because we do almost
the same as needed here to reorder display strings.

> > Amount of work is the least of my concerns in this regard.  I'm
> > worried about the effect the temporarily incorrect display will have
> > on users.
> 
> I don't recall the last time I ran into a problem with deferred
> font-lock.  I seriously doubt this is a real issue.

??? Set jit-lock-defer-time to something like 2, then visit xdisp.c,
scroll, and watch the fun.

It's not a "problem", deferred font-lock is _supposed_ to work like
that.  Except that what is tolerable for font-lock might not be for
display reordering.

Re: bidi-display-reordering is now non-nil by default

[Chong Yidong]

Eli Zaretskii <eliz@gnu.org> writes:

> What's wrong with using the `display' property with a special value,
> as I suggested a few mails ago?  It _is_ a character property, isn't
> it?

I'll have to think more about it, but I am not sure it's a good idea.
The display property is already used for a lot of other unrelated
purposes, like variable-width spaces.  So if you want to mark out a bidi
segment on a stretch of text that already has other display properties
present, you'd currently have to do some nasty merging of display specs.
Searching for the bidi display spec would be similarly annoying.  It
would be much nicer to have a separate text property specifically for
the purpose of marking out a bidi segment.

>> I don't recall the last time I ran into a problem with deferred
>> font-lock.  I seriously doubt this is a real issue.
>
> ??? Set jit-lock-defer-time to something like 2, then visit xdisp.c,
> scroll, and watch the fun.

Then don't do that.

Obviously, there is a trade-off: exploiting font-lock means more
"magically correct" segmenting; using a hack (like marking chars
special) means less dependence on the font-lock machinery.  I don't
think the latter is worth it.

Re: bidi-display-reordering is now non-nil by default

[Stefan Monnier]

> Obviously, there is a trade-off: exploiting font-lock means more
> "magically correct" segmenting; using a hack (like marking chars
> special) means less dependence on the font-lock machinery.  I don't
> think the latter is worth it.

The problem is not jit-lock-defer-time but the fact that font-lock may
be turned off.  We could do it in syntax-propertize, of course, but it's
not great either (e.g. syntax-propertize is applied to all the text
before window-start whether that text will ever be displayed or not).


        Stefan

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
> Date: Thu, 18 Aug 2011 23:16:13 -0400
> 
> > Obviously, there is a trade-off: exploiting font-lock means more
> > "magically correct" segmenting; using a hack (like marking chars
> > special) means less dependence on the font-lock machinery.  I don't
> > think the latter is worth it.
> 
> The problem is not jit-lock-defer-time but the fact that font-lock may
> be turned off.

Right, I mentioned that as well.

> We could do it in syntax-propertize, of course, but it's not great
> either (e.g. syntax-propertize is applied to all the text before
> window-start whether that text will ever be displayed or not).

So are you saying that using text properties for this is not a good
idea at all?  If so, what other Emacs features do we have to mark
segments of text for "special" reordering?  Markers, perhaps?

I think we should also think how will we use this feature if it
existed, before we go out to implement the feature.  What specific
segments of text we would want to mark, and in what kinds of
buffers/strings?  What mechanisms can be used to compute where to put
these markers, and how these mechanisms will be triggered?  How
exactly to handle HTML/XML with such a feature?  Etc., etc. -- all of
these questions can have a profound effect on the details of the
implementation, or even show that the idea is stillborn.

Re: bidi-display-reordering is now non-nil by default

[Chong Yidong]

Eli Zaretskii <eliz@gnu.org> writes:

> I think we should also think how will we use this feature if it
> existed, before we go out to implement the feature.  What specific
> segments of text we would want to mark, and in what kinds of
> buffers/strings?

Here is one example: in Emacs Lisp source code, I want every Lisp string
to be treated as a single bidi segment (paragraph).  Whichever way this
is done, these segments ought to correspond well (or exactly) to what
font-lock-mode marks with font-lock-string-face.  In particular, the
bidi segmentation should not be confused by stuff like escaped \" chars.

Re: bidi-display-reordering is now non-nil by default

[Chong Yidong]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Obviously, there is a trade-off: exploiting font-lock means more
>> "magically correct" segmenting; using a hack (like marking chars
>> special) means less dependence on the font-lock machinery.  I don't
>> think the latter is worth it.
>
> The problem is not jit-lock-defer-time but the fact that font-lock may
> be turned off.  We could do it in syntax-propertize, of course, but it's
> not great either (e.g. syntax-propertize is applied to all the text
> before window-start whether that text will ever be displayed or not).

I don't think we have to be concerned with the set of users who (i) wish
to turn off font-lock and (ii) edit source code with RTL characters and
(iii) expect those RTL segments to be automagically segmented according
to the programming language syntax.  Relax any one of those assumptions
and things are fine.

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Chong Yidong <cyd@stupidchicken.com>
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> Date: Thu, 18 Aug 2011 18:44:44 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > What's wrong with using the `display' property with a special value,
> > as I suggested a few mails ago?  It _is_ a character property, isn't
> > it?
> 
> I'll have to think more about it, but I am not sure it's a good idea.
> The display property is already used for a lot of other unrelated
> purposes, like variable-width spaces.  So if you want to mark out a bidi
> segment on a stretch of text that already has other display properties
> present, you'd currently have to do some nasty merging of display specs.

I don't see any need for any merging.  We already process display
specs on before- and after-strings, and the same code can be used
here.  IOW, those other display specs will be simply processed as part
of displaying the stretch of text covered by the bidi display spec.
Since the iterator state is pushed onto the stack, I see no issues
here that need merging.  If I'm missing something, please point out
what that is or show an example where we would need to merge display
specs.

> Searching for the bidi display spec would be similarly annoying.

We already do search for display properties as part of bidi scanning,
see bidi_fetch_char and compute_display_string_pos.  It would be a
simple job to add another spec to the list of those considered
relevant by handle_display_spec, which compute_display_string_pos
calls to see if the display spec needs special handling by the bidi
iteration.  What's left is to write a next_element_from_buffer_segment
or some such that is almost an identical twin of
next_element_from_string, and a few lines of code that initialize the
bidi iterator for reordering the segment, again almost like what we
already have for display strings.

> It would be much nicer to have a separate text property specifically
> for the purpose of marking out a bidi segment.

We can do that, too, but searching for 2 different properties is more
expensive than searching for just one.

Also, it isn't clear to me how your suggestion will work if the buffer
whose segments are determined by the bidi property is also reordered
"normally", e.g. if the surrounding text includes R2L characters.  The
non-linear iteration through the surrounding text will complicate the
way we take notice of the segments that are to be reordered
separately.

> >> I don't recall the last time I ran into a problem with deferred
> >> font-lock.  I seriously doubt this is a real issue.
> >
> > ??? Set jit-lock-defer-time to something like 2, then visit xdisp.c,
> > scroll, and watch the fun.
> 
> Then don't do that.

Users currently customize font-lock for purposes that have nothing to
do with bidi reordering.  With your suggestion, they will have two
contradicting goals for a single customization.  I don't see how they
will succeed in reconciling them.

Re: bidi-display-reordering is now non-nil by default

[Chong Yidong]

Eli Zaretskii <eliz@gnu.org> writes:

>> I'll have to think more about it, but I am not sure it's a good idea.
>> The display property is already used for a lot of other unrelated
>> purposes, like variable-width spaces.  So if you want to mark out a bidi
>> segment on a stretch of text that already has other display properties
>> present, you'd currently have to do some nasty merging of display specs.
>
> I don't see any need for any merging.  We already process display
> specs on before- and after-strings, and the same code can be used
> here.  IOW, those other display specs will be simply processed as part
> of displaying the stretch of text covered by the bidi display spec.

Suppose you have a stretch of text which includes characters with no
text properties plus a space with the display property

   '(space :width 3.0)

Suppose you want to use a display text property to mark that stretch of
text as a single bidi segment.  You'd add '(bidi-foo blahblah) to most
of the characters, but the property on the space character would have to
be something like

   '((bidi-foo blahblah) (space :width 3.0))

to avoid clobbering the pre-existing space display spec.

Obviously it is possible to do this, but it is just way cleaner and more
user-friendly to use a separate property for this purpose.

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Chong Yidong <cyd@stupidchicken.com>
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> Date: Fri, 19 Aug 2011 15:43:38 -0400
> 
>    '((bidi-foo blahblah) (space :width 3.0))
> 
> to avoid clobbering the pre-existing space display spec.
> 
> Obviously it is possible to do this, but it is just way cleaner and more
> user-friendly to use a separate property for this purpose.

If that's what you meant by "merging", then I simply misunderstood,
sorry.

However, handle_display_spec already handles such Lisp data structures
in display properties, so it's quite possible only minimum or no
changes will be needed to support this.  The only inconvenience is for
the Lisp code that would need to be more complicated when it puts
these properties.

Having a separate property has a disadvantage of being more expensive,
so unless someone implements a way of searching for a specified set of
properties, not just one or any one, I would prefer not to do that.
Especially since the plausibility of having other display properties
on the R2L segments is quite low, IMO.

Re: bidi-display-reordering is now non-nil by default

[Lars Magne Ingebrigtsen]

Chong Yidong <cyd@stupidchicken.com> writes:

> It would be much nicer to have a separate text property specifically
> for the purpose of marking out a bidi segment.

Obviously I know nothing about any of this, so I'm just talking as
someone who would like to use this stuff without knowing anything much
about what Emacs does internally.

To me, a nice interface to create a bit of text (that may later be
chopped up before insertion) would be to have a function like
(bidi-make-separate-string "lala").  This could perhaps be implemented
like (propertize "lala" 'bidi-region (make-unique-identifier)).

That is, what I'm saying is "make a string that will be treated
independently for bidi purposes".  If I insert two such strings next to
each other, I want them to be separate, even if they happen to have the
same l2r/r2l properties, so there would be no merging of regions.

I think.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Date: Fri, 19 Aug 2011 16:51:03 +0200
> 
> To me, a nice interface to create a bit of text (that may later be
> chopped up before insertion) would be to have a function like
> (bidi-make-separate-string "lala").  This could perhaps be implemented
> like (propertize "lala" 'bidi-region (make-unique-identifier)).
> 
> That is, what I'm saying is "make a string that will be treated
> independently for bidi purposes".

Is using the directional control characters and/or TABs permitted for
the solution?  If it is, there's more than one way to do what you
want, I will show you the most bulletproof one as soon as you agree to
see it ;-)

If using directional controls is not allowed, or frowned upon, then we
currently lack the infrastructure to do that, and as you see from this
thread, exactly which feature would be best for doing this is still
being debated.

Btw, I'm not at all sure the API you suggest is the best one.  One
alternative would be to specify the entire line, with marker
characters to show separate fields, and ask to modify this line such
that fields will be displayed left to right (or right to left) in
their order in the string.  It's possible there are other alternatives
as well.

Re: bidi-display-reordering is now non-nil by default

[Andrey Paramonov]

On 15.08.2011 13:10, Eli Zaretskii wrote:
>> Date: Mon, 15 Aug 2011 12:04:25 +0400
>> From: Andrey Paramonov<paramon@acdlabs.ru>
>>
>> However if the special direction symbols will start to leak into some
>> "usual" buffers, like programming language/xml modes, that might become
>> a problem. Now if an UTF-8 buffer contains only latin characters, it's
>> also ASCII. This is very common for a program source to be ASCII-only.
>> But if special symbols are added to it by Emacs, it's no more ASCII and
>> there is no guarantee that is will then correctly be compiled by another
>> program.
>>
>> Just an opinion from a L2R user here.
>
> Thanks.
>
> Emacs should not (and does not) insert directional controls into any
> "usual" buffers, unless the user inserts them manually, or asks for
> that in some other way.

It's important Emacs continues to work this way, for "usual" buffers.

 > These control characters can "leak" into such
> buffers only if the user copy/pastes text from some "unusual" buffer.
> But the same issue will happen if one pastes text with some other
> non-ASCII character, like Cyrillic.

This is perfectly fine.

Best wishes,
Andrey Paramonov

Re: bidi-display-reordering is now non-nil by default

[Andrey Paramonov]

 > If a comment or a string in a source file use R2L scripts, how
 > would you like Emacs to display them?

Frankly, I doubt that there is a single source file with R2L comments in 
the world, let alone R2L identifiers. That would be funny to read ;-) 
Even Cyrillic comments in source code are not that common.

Unlike "special" and plain-text buffers, the bidirectionalization of 
programming modes just seems to have very little to no value right now.

Best wishes,
Andrey Paramonov

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> Date: Tue, 16 Aug 2011 14:02:49 +0400
> From: Andrey Paramonov <paramon@acdlabs.ru>
> 
>  > If a comment or a string in a source file use R2L scripts, how
>  > would you like Emacs to display them?
> 
> Frankly, I doubt that there is a single source file with R2L comments in 
> the world, let alone R2L identifiers.

I wasn't talking about identifiers.

I don't know how many comments in R2L languages are out there, but I
hope you will agree that strings with R2L text are a definite
possibility, and should be supported.

> Unlike "special" and plain-text buffers, the bidirectionalization of 
> programming modes just seems to have very little to no value right now.

Which is why the relevant support for them was not the 1st priority.
But that doesn't mean we need to dismiss it as something to be
supported in the future.

Re: bidi-display-reordering is now non-nil by default

[Andrey Paramonov]

On 16.08.2011 14:40, Eli Zaretskii wrote:
>> Date: Tue, 16 Aug 2011 14:02:49 +0400
>> From: Andrey Paramonov<paramon@acdlabs.ru>
>>
>>   >  If a comment or a string in a source file use R2L scripts, how
>>   >  would you like Emacs to display them?
>>
>> Frankly, I doubt that there is a single source file with R2L comments in
>> the world, let alone R2L identifiers.
>
> I wasn't talking about identifiers.
>
> I don't know how many comments in R2L languages are out there, but I
> hope you will agree that strings with R2L text are a definite
> possibility, and should be supported.
>

Sorry, I misread "strings" for "identifiers". Strings are important indeed.

>> Unlike "special" and plain-text buffers, the bidirectionalization of
>> programming modes just seems to have very little to no value right now.
>
> Which is why the relevant support for them was not the 1st priority.
> But that doesn't mean we need to dismiss it as something to be
> supported in the future.
>

Sure.

Best wishes,
Andrey Paramonov