Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii <eliz@gnu.org>]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Wed, 24 Aug 2011 10:51:34 -0400
> 
> >> Switch to buffer: a{data.tex,foo.ada,*scratch*}
> >> so maybe the {,,,} list should use bidi-string-mark-left-to-right before
> >> the comma.
> > They should probably run each completion candidate through
> > bidi-string-mark-left-to-right, yes.
> 
> That was my impression as well.
> 
> > I say "probably" because there's a separate but related issue with
> > buffer and file names that include R2L characters and end in weak or
> > neutral characters: those weak or neutrals will be displayed to the
> > left of the R2L characters, as in "DCBA!" (instead of the more
> > plausible "!DCBA"), and appending the LRM will not cure that.
> 
> I'm a bit confused here: you say "to the left of the R2L characters",
> but your example shows it to the right.

Yes, I meant the other left ;-)

> This said, it seems the problem you're referring to is a lot more
> general than to icomplete/ido/iswitchb: the same problem will show up
> wherever the file/buffer is displayed, no?

Yes, but with buffer/file contents it's a different story.  Whoever
writes the buffer/file text can (and should) use the directional
controls or accept the consequences.

By contrast, what Emacs shows in the echo area, the minibuffer, and
the buffers it pops up, like *Completions*, is something Emacs itself
generates, and the user has no control on those contents.  So we
should have some general policy in place, and have supporting
infrastructure, that would allow a Lisp program to DTRT with names of
files, buffers, etc. that we show in these contexts.

For example, when we prompt for something and show the default value,
the presence of R2L characters in that default should not produce the
kind of jumbled display we sometimes show with the current codebase.
Here's a simple example:

 emacs -Q
 M-: (read-buffer "Buffer name: " "אבגדה") RET
 M-n

> > It is also possible we should add a more general bidi-string-mark-dwim
> > function, which uses either LRM or RLM depending on the paragraph
> > direction, to avoid additional changes later, when the UI will be
> > localizable.
> 
> Yes, that's what I was getting at, basically I think that we almost
> never want to add a LRM but instead we want to add some kind of
> segment-separator.

It turns out a segment separator (in the UAX#9 definition, i.e. a TAB)
does not always help.  Here, for example, it does not, because the
colon and the closing parenthesis interfere:

     Buffer name (default אבגד):	12345

Remove the colon and the closing paren, and everything is dandy:

     Buffer name (default אבגד	12345

This is a strange behavior, it caught me by surprise.  But this
behavior is correct, as confirmed by Yudit and by the UAX#9 Reference
Implementation, so it's not a bug.  It's what the UBA mandates.

It looks like the only solution that always works is to use the
Paragraph Separator character (u+2029).  That's because each paragraph
is reordered separately and independently.

> E.g. I think that a buffer ABC<1> should be either displayed as CBA<1>
> or <1>CBA depending on the surrounding text direction (which might be
> different from the paragraph direction).

Not sure what you mean here.  Do you mean reorder that as a single
atomic unit?  If not, what do you mean by "surrounding text
direction"?  The surrounding text is different for each character of
the "<1>" string, and what we do with it by default is precisely
display depending on the surrounding text direction.

> > > > Need your decision on the space spec.
> > > Is the above enough, or are there still more open issues?
> > If the above is a "go ahead" for 24.1, then no open issues left.
> 
> Yes, it's a "go ahead" for the "treat any `space' display property as
> a segment separator".

Well, in light of the above, it sounds like we need to treat such
properties as a paragraph separator, to get the effect we need.  I'll
take a stab at that.