Re: bidi-display-reordering is now non-nil by default

Re: bidi-display-reordering is now non-nil by default

[Andrey Paramonov]

 > I sometimes get prompted to choose a different encoding when I
 > send email responses to mail that looked pure ASCII. I'm
 > presented with a buffer that points to a character that doesn't
 > look "foreign" at all. If I care to investigate, it turns out
 > that character was an em-dash or fancy quotes, that just
 > looked "normal" due to the font. Am I confused? No. I just
 > select the encoding, usually the one suggested by Emacs as the
 > default, and that's it.

If the insertion of special symbols can make the behavior of "special" 
buffers (like dired, Gnus summary etc) robust/correct, I see no problem 
in doing so always, not only in presence of R2L text. The prompt 
described above can only appear if one doesn't use Unicode and wants to 
save a special buffer (like dired) to a file. The combination of such 
events seems extremely rare (although not improbable, of course).

However if the special direction symbols will start to leak into some 
"usual" buffers, like programming language/xml modes, that might become 
a problem. Now if an UTF-8 buffer contains only latin characters, it's 
also ASCII. This is very common for a program source to be ASCII-only. 
But if special symbols are added to it by Emacs, it's no more ASCII and 
there is no guarantee that is will then correctly be compiled by another 
program.

Just an opinion from a L2R user here.

Best wishes,
Andrey Paramonov

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> Date: Mon, 15 Aug 2011 12:04:25 +0400
> From: Andrey Paramonov <paramon@acdlabs.ru>
> 
> However if the special direction symbols will start to leak into some 
> "usual" buffers, like programming language/xml modes, that might become 
> a problem. Now if an UTF-8 buffer contains only latin characters, it's 
> also ASCII. This is very common for a program source to be ASCII-only. 
> But if special symbols are added to it by Emacs, it's no more ASCII and 
> there is no guarantee that is will then correctly be compiled by another 
> program.
> 
> Just an opinion from a L2R user here.

Thanks.

Emacs should not (and does not) insert directional controls into any
"usual" buffers, unless the user inserts them manually, or asks for
that in some other way.  These control characters can "leak" into such
buffers only if the user copy/pastes text from some "unusual" buffer.
But the same issue will happen if one pastes text with some other
non-ASCII character, like Cyrillic.

Re: bidi-display-reordering is now non-nil by default

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Mon, 15 Aug 2011 12:04:25 +0400
>> From: Andrey Paramonov <paramon@acdlabs.ru>
>> 
>> However if the special direction symbols will start to leak into some 
>> "usual" buffers, like programming language/xml modes, that might become 
>> a problem. Now if an UTF-8 buffer contains only latin characters, it's 
>> also ASCII. This is very common for a program source to be ASCII-only. 
>> But if special symbols are added to it by Emacs, it's no more ASCII and 
>> there is no guarantee that is will then correctly be compiled by another 
>> program.
>> 
>> Just an opinion from a L2R user here.
>
> Thanks.
>
> Emacs should not (and does not) insert directional controls into any
> "usual" buffers, unless the user inserts them manually, or asks for
> that in some other way.  These control characters can "leak" into such
> buffers only if the user copy/pastes text from some "unusual" buffer.
> But the same issue will happen if one pastes text with some other
> non-ASCII character, like Cyrillic.

I can still pick iso-8869-5 encoding for a cyrillic cut&paste, but not
if it contains L2R marks.  And when having pure ASCII with L2R mark
leakage into it, I can't save under any 8-bit encoding (well, iso-8859-8
would work, but who is going to be able to read it?).

Emacs should really avoid formatting things with L2R marks that are not
actually required.

Is there a reason nobody responded to my repeated proposal to let
`format' deal with inserting L2R marks?

-- 
David Kastrup

Re: bidi-display-reordering is now non-nil by default

[Andrey Paramonov]

On 15.08.2011 13:10, Eli Zaretskii wrote:
>> Date: Mon, 15 Aug 2011 12:04:25 +0400
>> From: Andrey Paramonov<paramon@acdlabs.ru>
>>
>> However if the special direction symbols will start to leak into some
>> "usual" buffers, like programming language/xml modes, that might become
>> a problem. Now if an UTF-8 buffer contains only latin characters, it's
>> also ASCII. This is very common for a program source to be ASCII-only.
>> But if special symbols are added to it by Emacs, it's no more ASCII and
>> there is no guarantee that is will then correctly be compiled by another
>> program.
>>
>> Just an opinion from a L2R user here.
>
> Thanks.
>
> Emacs should not (and does not) insert directional controls into any
> "usual" buffers, unless the user inserts them manually, or asks for
> that in some other way.

It's important Emacs continues to work this way, for "usual" buffers.

 > These control characters can "leak" into such
> buffers only if the user copy/pastes text from some "unusual" buffer.
> But the same issue will happen if one pastes text with some other
> non-ASCII character, like Cyrillic.

This is perfectly fine.

Best wishes,
Andrey Paramonov

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Mon, 15 Aug 2011 11:24:04 +0200
> 
> I can still pick iso-8869-5 encoding for a cyrillic cut&paste, but not
> if it contains L2R marks.

How is having Cyrillic characters different from having an LRM?

> And when having pure ASCII with L2R mark leakage into it, I can't
> save under any 8-bit encoding (well, iso-8859-8 would work, but who
> is going to be able to read it?).

ASCII characters are not changed by iso-8859-8 in any way, and LRM
doesn't need to be "read" at all.

> Emacs should really avoid formatting things with L2R marks that are not
> actually required.

You are welcome to code a function that determines when it is actually
required.  One problem with doing so with 100% accuracy is that the
outcome depends on the text that follows the string in question, and
sometimes even on text that precedes it.  These are not always known,
and in fact the text that follows will normally be unknown.

Re: bidi-display-reordering is now non-nil by default

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>
>> Emacs should really avoid formatting things with L2R marks that are not
>> actually required.
>
> You are welcome to code a function that determines when it is actually
> required.  One problem with doing so with 100% accuracy is that the
> outcome depends on the text that follows the string in question, and
> sometimes even on text that precedes it.  These are not always known,
> and in fact the text that follows will normally be unknown.

Your stance is that it is ok to pepper text prepared by Emacs (and the
Emacs sources all across the board) with LRM characters as long as it
has a moderate chance of getting closer to 100% accuracy _when_ Hebrew
characters move into the payload.

But you don't get that accuracy anyway since it _only_ works for those
areas that are explicitly prepared for it, and external maintainers of
modes will not change all their code on the theoretic assumption that
somebody might get ugly results when injecting Hebrew.

We have been that down the "everybody will adapt if forced to properly"
road already with multibyte characters in 20.2 or so, when
(forward-char) was different from (goto-char (1+ (point))).  That
approach failed and was replaced with a different, less efficient solution.

Any solution that requires code review and adaptation everywhere in code
that works fine without R2L content is not going to work.

Blind activism is not going to solve the problem you perceive.

-- 
David Kastrup

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Mon, 15 Aug 2011 12:46:09 +0200
> 
> Your stance is that it is ok to pepper text prepared by Emacs (and the
> Emacs sources all across the board) with LRM characters as long as it
> has a moderate chance of getting closer to 100% accuracy _when_ Hebrew
> characters move into the payload.
> 
> But you don't get that accuracy anyway since it _only_ works for those
> areas that are explicitly prepared for it, and external maintainers of
> modes will not change all their code on the theoretic assumption that
> somebody might get ugly results when injecting Hebrew.
> 
> We have been that down the "everybody will adapt if forced to properly"
> road already with multibyte characters in 20.2 or so, when
> (forward-char) was different from (goto-char (1+ (point))).  That
> approach failed and was replaced with a different, less efficient solution.
> 
> Any solution that requires code review and adaptation everywhere in code
> that works fine without R2L content is not going to work.
> 
> Blind activism is not going to solve the problem you perceive.

You are dead wrong: I have no activism stance in this matter.  I
actually don't care.  My job was to give Emacs infrastructure upon
which bidi-aware features could be built.  That job is almost done,
modulo bugs that will show during the pretest.  The rest is the
responsibility of those unnamed "external maintainers of modes"--they
are free not to care about catering to the hundreds of millions of
readers of R2L scripts enough to adapt: it's their funeral.  I myself
rarely if at all use any R2L scripts in Emacs.

Everything I wrote in this thread is just advice, no less, no more.
Advice from someone who knows something about how reordering works,
and what can or cannot be easily solved or coded.  You can take it or
you can continue attacking it, be my guest any way.  But if you want
to convince me, you will have to come up with something more tangible
than just FUD.

Re: bidi-display-reordering is now non-nil by default

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

> You are dead wrong: I have no activism stance in this matter.  I
> actually don't care.  My job was to give Emacs infrastructure upon
> which bidi-aware features could be built.  That job is almost done,
> modulo bugs that will show during the pretest.

And that is an important job that makes Emacs usable for a new audience.
It won't make it pretty for the new audience, but Emacs is not really
pretty except for users comfortable with an English interface.

> The rest is the responsibility of those unnamed "external maintainers
> of modes"--they are free not to care about catering to the hundreds of
> millions of readers of R2L scripts enough to adapt: it's their
> funeral.

Nope, it is Emacs' funeral if one can't expect it to produce consistent
results without lots of changes all across third party code bases.

My position is that we should be quite conservative with changing its
behavior for pure L2R material, or requiring such changes from
third-party code authors.

> Everything I wrote in this thread is just advice, no less, no more.
> Advice from someone who knows something about how reordering works,
> and what can or cannot be easily solved or coded.  You can take it or
> you can continue attacking it, be my guest any way.  But if you want
> to convince me, you will have to come up with something more tangible
> than just FUD.

There is no way that I can make you acknowledge something you choose to
ignore.  But since you are not the only Emacs developer, raising the
issues may still make them register with others.

-- 
David Kastrup

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Mon, 15 Aug 2011 13:27:47 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > You are dead wrong: I have no activism stance in this matter.  I
> > actually don't care.  My job was to give Emacs infrastructure upon
> > which bidi-aware features could be built.  That job is almost done,
> > modulo bugs that will show during the pretest.
> 
> And that is an important job that makes Emacs usable for a new audience.
> It won't make it pretty for the new audience, but Emacs is not really
> pretty except for users comfortable with an English interface.

There's nothing in Emacs yet to present an interface that is not
English.  I actually considered coding a couple of features to make it
possible for the UI to be displayed right-to-left, but eventually
decided against it, because it didn't make sense to do that for Emacs
that doesn't support localized UI at all.  (There are comments in the
code and a TODO item for someone who will want to implement UI
direction control in the future.)

However, what we are talking about here is not the interface, it's the
ability to edit bidirectional text.  Typing non-English text in Emacs
is dead simple, for quite some time, as simple as in any other
application on the same platform.  It is as "pretty" as it gets.

> > The rest is the responsibility of those unnamed "external maintainers
> > of modes"--they are free not to care about catering to the hundreds of
> > millions of readers of R2L scripts enough to adapt: it's their
> > funeral.
> 
> Nope, it is Emacs' funeral if one can't expect it to produce consistent
> results without lots of changes all across third party code bases.

If you think support for bidirectional scripts can come without "lots
of changes" where it matters, you are in the pipe dream land.  Sorry,
that's life.  Look how any bidi-aware HTML needs to use the equivalent
directives in order to DTRT--there's a lesson here to be learned.

> My position is that we should be quite conservative with changing its
> behavior for pure L2R material, or requiring such changes from
> third-party code authors.

If we can do that, fine.  The design and implementation of the bidi
infrastructure followed this principle from day one.  But sometimes,
there's nothing you can do except change the code in higher levels.
After all, breaking the assumption that "before" in the buffer means
"to the left" on the screen is an extremely fundamental change.  Any
code that depends on that assumption will need to do _something_ to
produce legible display.

Now, if someone can show a way to fix this use case in Emacs
infrastructure such as `format', I'm all ears.  You raised the idea,
but never explained it in enough details to judge it.  However, from
what I gather, your suggestion means changes in 3rd party code anyway
(e.g., use a special format specifier).

> There is no way that I can make you acknowledge something you choose
> to ignore.

Your views are not ignored, they were carefully considered and
rejected.

> But since you are not the only Emacs developer, raising the issues
> may still make them register with others.

I would be more than thrilled to see someone to come on board and
help.

Re: bidi-display-reordering is now non-nil by default

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

> However, what we are talking about here is not the interface, it's the
> ability to edit bidirectional text.

It would make discussion more productive if you actually remembered what
you are getting defensive about.

The point of discussion was the display (and preparation) of mode lines
and prompts as one example of Emacs-prepared text, going to special
buffer displays like dired, and whether we want Emacs to insert
directional marks in content like that even when no R2L material is
present.

The ability to edit bidirectional text is not affected either way by
Emacs inserting directional marks into strings it generates.

>> But since you are not the only Emacs developer, raising the issues
>> may still make them register with others.
>
> I would be more than thrilled to see someone to come on board and
> help.

Making Emacs annoying to L2R users might be one way to capture their
attention.  I can't deny it worked for multibyte support.  But I am not
fond of that approach.

-- 
David Kastrup

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Mon, 15 Aug 2011 14:56:50 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > However, what we are talking about here is not the interface, it's the
> > ability to edit bidirectional text.
> 
> It would make discussion more productive if you actually remembered what
> you are getting defensive about.

Sorry, my memory is limited.  This has been a very long thread.

> The point of discussion was the display (and preparation) of mode lines
> and prompts as one example of Emacs-prepared text, going to special
> buffer displays like dired

The main issue that started this was the Gnus summary and similar
buffers, which present table-like summary of a list of items.  Dired
was barely mentioned, and I don't remember prompts or mode lines
mentioned, except as in passage.  Please remind me what was the
problem with the mode line.

> and whether we want Emacs to insert directional marks in content
> like that even when no R2L material is present.

Yes.  To reiterate, I think doing this unconditionally is the easiest
way, since any kind of conditions will not be able to be 100% accurate
without lots of complications for the users of the relevant APIs.
It's quite possible that 100% accuracy will be entirely impractical.

> Making Emacs annoying to L2R users might be one way to capture their
> attention.  I can't deny it worked for multibyte support.  But I am not
> fond of that approach.

I have no idea what is this about.

Re: bidi-display-reordering is now non-nil by default

[Stefan Monnier]

> Emacs should really avoid formatting things with L2R marks that are not
> actually required.

And then

> Is there a reason nobody responded to my repeated proposal to let
> `format' deal with inserting L2R marks?

AFAICT these two are contradictory.  To only insert them when they are
really needed, we need additional hints that only a human can give.
Hence the new function.  Doing it in format would risk doing it in many
more cases where they're not actually indispensable.

But in any case, the bidi support in Emacs-24.1 is just a first step.
Clearly we'll need more work to make it work well.  The most obvious
case being R2L strings within programming and markup languages: adding
LRM marks in them would be needed to have them display correctly, but
at the same time we really don't want to add those LRM marks in the
corresponding file (nor in copy&paste text).

I think arguing about how best to solve problems in gnus-sum is futile
until we've figured out how to solve the problem for
markup&programming modes.
And in the mean time, using string-mark-left-to-right seems like
a fine stopgap.


        Stefan

Re: bidi-display-reordering is now non-nil by default

[David Kastrup]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Emacs should really avoid formatting things with L2R marks that are not
>> actually required.
>
> And then
>
>> Is there a reason nobody responded to my repeated proposal to let
>> `format' deal with inserting L2R marks?
>
> AFAICT these two are contradictory.

If "format" creates a pure L2R string, it will not insert directional
marks.  That is rather simple.

When it puts stuff together that would make cause changes of
directionality from inside a field to outside, it would insert marks to
avoid this.  So format("%sx", v) would not be equivalent to concat(v,
"x").  Something like format("%\\sx", v) might: one would try to make
some modifier for "don't insert directional marks".

> To only insert them when they are really needed, we need additional
> hints that only a human can give.

To get a hit rate of 100%, sure, nothing but human intervention will do.
But if the point is to get behavior that is rather close to optimal
without regressions for L2R texts and without human intervention
everywhere, addressing format is going to give a good start.  Where is
the sense in fields like %-10.10s" if format is going to cut off a
directional mark the user added for proper display?  If format is the
entity inserting the mark in the first place, at least this will go less
wrong.

> But in any case, the bidi support in Emacs-24.1 is just a first step.
> Clearly we'll need more work to make it work well.

Nothing wrong with that.  I'd like to avoid a situation where we'll need
to backpaddle to make it work well.

-- 
David Kastrup

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Mon, 15 Aug 2011 09:59:34 -0400
> Cc: emacs-devel@gnu.org
> 
> I think arguing about how best to solve problems in gnus-sum is futile
> until we've figured out how to solve the problem for
> markup&programming modes.

I don't think so.  The former is a much simpler problem, because it
has to do with text that normally isn't edited nor submitted to
external programs.  It is basically a problem of concatenating strings
subject to certain restrictions wrt their layout on the screen, like
"make str1 display to the left of str2".

The latter problem is much harder, because it needs a capability to
reorder only certain portions of a buffer.

Re: bidi-display-reordering is now non-nil by default

[Stefan Monnier]

> Nothing wrong with that.  I'd like to avoid a situation where we'll need
> to backpaddle to make it work well.

The only special thing we have currently installed is
string-mark-left-to-right which is not only a trivial functionality
that's easy to support in the future, but whose uses are also trivial to
find if we come up with something better in the future.

So, I think it will do fine.  Feel free to experiment with a patch to
`format' which we might even include in 24.2, but that's too risky at
this point.


        Stefan

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Mon, 15 Aug 2011 16:18:14 +0200
> 
> If "format" creates a pure L2R string, it will not insert directional
> marks.

What is a "pure L2R string", for this purpose?  Is this a string that
does not get reordered for display, or a string that does not affect
strings concatenated with it?  Or something else?  I think this should
be carefully defined to make sense in this context; each of the above
alternatives could yield a different result for the same string.

> When it puts stuff together that would make cause changes of
> directionality from inside a field to outside, it would insert marks to
> avoid this.  So format("%sx", v) would not be equivalent to concat(v,
> "x").  Something like format("%\\sx", v) might: one would try to make
> some modifier for "don't insert directional marks".

This might work (although is non-trivial to code), but it requires
that the following string(s) be known to `format'.  Note: not just the
single following substring, but as much text as is needed to determine
whether a simple concatenation will or will not work.  For example, if
"x" in the above example is a string made of weak or neutral
characters (e.g., digits or punctuation or whitespace), `format' will
be unable to DTRT, unless we accept the sub-optimal solution of
inserting directional controls in this case anyway.

Of course, it also means relevant Lisp code will have to be modified
to not use `concat' or `insert' to generate text out of its fragments,
but instead to use these special features of `format'.

> But if the point is to get behavior that is rather close to optimal
> without regressions for L2R texts

This "rather close to optimal" goal should also be clearly defined,
IMO.

Re: bidi-display-reordering is now non-nil by default

[Stefan Monnier]

>> I think arguing about how best to solve problems in gnus-sum is futile
>> until we've figured out how to solve the problem for
>> markup&programming modes.
> I don't think so.  The former is a much simpler problem, because it
> has to do with text that normally isn't edited nor submitted to
> external programs.  It is basically a problem of concatenating strings
> subject to certain restrictions wrt their layout on the screen, like
> "make str1 display to the left of str2".
> The latter problem is much harder, because it needs a capability to
> reorder only certain portions of a buffer.

We agree.  I just think that once we've solved the hard problem, the
best solution for the simple problem will probably be quite different
from what we would have come up with at first.

And currently string-mark-left-to-right does the job just fine.


        Stefan

Re: bidi-display-reordering is now non-nil by default

[Chong Yidong]

Eli Zaretskii <eliz@gnu.org> writes:

> I don't think so.  The former is a much simpler problem, because it
> has to do with text that normally isn't edited nor submitted to
> external programs.  It is basically a problem of concatenating strings
> subject to certain restrictions wrt their layout on the screen, like
> "make str1 display to the left of str2".
>
> The latter problem is much harder, because it needs a capability to
> reorder only certain portions of a buffer.

Here's an idea: how bout if programming language and related buffers
employ a modified bidi type table in which punctuation characters
(like + " and <) are bidi segment separators (S)?

AFAICT, this hack would fix most of the garbaging problems in source
code as some others like "buffername<1>".

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Chong Yidong <cyd@stupidchicken.com>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, dak@gnu.org,
>         emacs-devel@gnu.org
> Date: Mon, 15 Aug 2011 14:28:23 -0400
> 
> Here's an idea: how bout if programming language and related buffers
> employ a modified bidi type table in which punctuation characters
> (like + " and <) are bidi segment separators (S)?

Currently, the bidi type table is global, so you cannot change it on a
buffer by buffer basis.

More importantly, I don't think it's a good idea to change the bidi
properties in such a radical way (or at all), because we don't have
the resources to investigate all of the consequences.  For example, <
is a mirrored character, so when it's between to R2L characters, it
displays as > and vice versa.  Yes, I know: you'll suggest to change
the mirroring table as well, but how far should we go? do we really
want to create our own UBA?

> AFAICT, this hack would fix most of the garbaging problems in source
> code as some others like "buffername<1>".

But it will do The Wrong Thing if < or > are parts of the buffer name
itself.

Re: bidi-display-reordering is now non-nil by default

[Stefan Monnier]

> More importantly, I don't think it's a good idea to change the bidi
> properties in such a radical way (or at all), because we don't have
> the resources to investigate all of the consequences.  For example, <
> is a mirrored character, so when it's between to R2L characters, it
> displays as > and vice versa.  Yes, I know: you'll suggest to change
> the mirroring table as well, but how far should we go? do we really
> want to create our own UBA?

Furthermore, this buffer-local table should only be used for actual
code, but the default global table should probably be used within
comments and strings.


        Stefan

Re: bidi-display-reordering is now non-nil by default

[Chong Yidong]

Eli Zaretskii <eliz@gnu.org> writes:

> More importantly, I don't think it's a good idea to change the bidi
> properties in such a radical way (or at all), because we don't have
> the resources to investigate all of the consequences.  For example, <
> is a mirrored character, so when it's between to R2L characters, it
> displays as > and vice versa.  Yes, I know: you'll suggest to change
> the mirroring table as well, but how far should we go? do we really
> want to create our own UBA?

The specs allow for a "higher level protocol" to segment text according
to what makes sense for (say) Lisp syntax, so anything that displays
most realistic Lisp code correctly would be good enough.

I guess this is not realistic to implement for 24.1, though, so since
there is no near-term solution, how bout turning off bidi display
reordering for prog-mode buffers?

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Chong Yidong <cyd@stupidchicken.com>
> Cc: monnier@iro.umontreal.ca, dak@gnu.org, emacs-devel@gnu.org
> Date: Mon, 15 Aug 2011 22:02:53 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > More importantly, I don't think it's a good idea to change the bidi
> > properties in such a radical way (or at all), because we don't have
> > the resources to investigate all of the consequences.  For example, <
> > is a mirrored character, so when it's between to R2L characters, it
> > displays as > and vice versa.  Yes, I know: you'll suggest to change
> > the mirroring table as well, but how far should we go? do we really
> > want to create our own UBA?
> 
> The specs allow for a "higher level protocol" to segment text according
> to what makes sense for (say) Lisp syntax, so anything that displays
> most realistic Lisp code correctly would be good enough.

"Higher level protocols" don't include futzing with bidirectional
properties of characters.  "Higher level protocol" means some means to
determine segment boundaries other than segment separator characters
that are part of the text.  Translated into Emacs-speak, this means we
need a variable that, when bound to some special value, instructs the
reordering engine to treat certain characters as segment separators.

> I guess this is not realistic to implement for 24.1

If we go with a variable described above, it's actually a very simple
job, so just say the word.  However, I suggest that before doing this
we come up with a more detailed plan: suppose you have this variable,
how would you solve the problems discussed here with its help?  The
devil is in the details.

> since there is no near-term solution, how bout turning off bidi
> display reordering for prog-mode buffers?

What for?  No one complained about it yet, and leaving it on helps
find bugs and inefficiencies in the bidi display engine.  So my vote
is NAY.

Re: bidi-display-reordering is now non-nil by default

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Chong Yidong <cyd@stupidchicken.com>
>
>> since there is no near-term solution, how bout turning off bidi
>> display reordering for prog-mode buffers?
>
> What for?  No one complained about it yet, and leaving it on helps
> find bugs and inefficiencies in the bidi display engine.  So my vote
> is NAY.

Sounds like the plan is to make Emacs painful to use for everyone in
order to make it a priority for L2R users to solve R2L problems.

That may be a pretest strategy.  But I don't think it is a good idea to
stay with it for the betas, or even the release.

-- 
David Kastrup

Re: bidi-display-reordering is now non-nil by default

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

> Translated into Emacs-speak, this means we need a variable that, when
> bound to some special value, instructs the reordering engine to treat
> certain characters as segment separators.

They already exist: comment-start and comment-end, for example.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: bidi-display-reordering is now non-nil by default

[David Kastrup]

Andreas Schwab <schwab@linux-m68k.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> Translated into Emacs-speak, this means we need a variable that, when
>> bound to some special value, instructs the reordering engine to treat
>> certain characters as segment separators.
>
> They already exist: comment-start and comment-end, for example.

I am having a somewhat hard time imagining it as a good thing if Emacs
displays source code that does no longer correspond to a reasonably
straightforward manner of printing the file.  reordering source into
something that "makes more sense" seems quite more invasive than mere
font-locking.

I am not fond of the idea of writing code that gets unreadable unless
you read it back with Emacs, or even a specific version of Emacs or
specific settings.

It may partly be the problem space of the language specification:
something like HTML should likely give ways of using of directional
marks as source formatting aids without affecting the content payload.

-- 
David Kastrup

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: Chong Yidong <cyd@stupidchicken.com>,  dak@gnu.org,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Tue, 16 Aug 2011 09:40:45 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Translated into Emacs-speak, this means we need a variable that, when
> > bound to some special value, instructs the reordering engine to treat
> > certain characters as segment separators.
> 
> They already exist: comment-start and comment-end, for example.

These are strings, not characters.  I meant single characters.
Making that a string would make the job of implementing such a
high-level protocol much harder, because the reordering engine always
considers only one character, with a very few exceptions.

Also, a segment cannot span more than a single line (up to a newline),
so multi-line comments cannot use this solution.

What I had in mind was not comments and strings in program source, but
issues with display of foobar<1> buffer names and lines in
mail-summary buffers, which I thought was what Chong had in mind.  It
could also be a solution for displaying XML, although that use case
needs more thorough analysis, to see if this would really be a good
solution.

The key issue here is whether those same characters can appear in
context where we do want them to be reordered as "normal" punctuation.
If the answer is YES, then this idea won't hold water.

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: Chong Yidong <cyd@stupidchicken.com>,  dak@gnu.org,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Tue, 16 Aug 2011 09:40:45 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Translated into Emacs-speak, this means we need a variable that, when
> > bound to some special value, instructs the reordering engine to treat
> > certain characters as segment separators.
> 
> They already exist: comment-start and comment-end, for example.

These are strings, not characters.  I meant single characters.
Making that a string would make the job of implementing such a
high-level protocol much harder, because the reordering engine always
considers only one character, with a very few exceptions.

Also, a segment cannot span more than a single line (up to a newline),
so multi-line comments cannot use this solution.

What I had in mind was not comments and strings in program source, but
issues with display of foobar<1> buffer names and lines in
mail-summary buffers, which I thought was what Chong had in mind.  It
could also be a solution for displaying XML, although that use case
needs more thorough analysis, to see if this would really be a good
solution.

The key issue here is whether those same characters can appear in
context where we do want them to be reordered as "normal" punctuation.
If the answer is YES, then this idea won't hold water.

Re: bidi-display-reordering is now non-nil by default

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andreas Schwab <schwab@linux-m68k.org>
>> Cc: Chong Yidong <cyd@stupidchicken.com>,  dak@gnu.org,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
>> Date: Tue, 16 Aug 2011 09:40:45 +0200
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > Translated into Emacs-speak, this means we need a variable that, when
>> > bound to some special value, instructs the reordering engine to treat
>> > certain characters as segment separators.
>> 
>> They already exist: comment-start and comment-end, for example.
>
> These are strings, not characters.  I meant single characters.

You can't have that.  The field separators are multi-character strings.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Tue, 16 Aug 2011 09:54:18 +0200
> 
> Andreas Schwab <schwab@linux-m68k.org> writes:
> 
> > Eli Zaretskii <eliz@gnu.org> writes:
> >
> >> Translated into Emacs-speak, this means we need a variable that, when
> >> bound to some special value, instructs the reordering engine to treat
> >> certain characters as segment separators.
> >
> > They already exist: comment-start and comment-end, for example.
> 
> I am having a somewhat hard time imagining it as a good thing if Emacs
> displays source code that does no longer correspond to a reasonably
> straightforward manner of printing the file.  reordering source into
> something that "makes more sense" seems quite more invasive than mere
> font-locking.

Please explain what you mean by "no longer corresponds".  If a comment
or a string in a source file use R2L scripts, how would you like Emacs
to display them?  If the answer is in their original logical order,
then how is the user staring at the source supposed to read those
comments and strings?

As for printing, don't modern printers and associated software already
reorder text when they print?  If not, we may be able to use Emacs
facilities to DTRT here.

> I am not fond of the idea of writing code that gets unreadable unless
> you read it back with Emacs, or even a specific version of Emacs or
> specific settings.

??? The discussion was about _displaying_, no one suggested writing
anything to the file.  The idea was to use _existing_ characters or
strings that delimit the relevant portions of source as a cue for
reordering those delimited parts in some way that makes them readable.

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Tue, 16 Aug 2011 09:07:16 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Chong Yidong <cyd@stupidchicken.com>
> >
> >> since there is no near-term solution, how bout turning off bidi
> >> display reordering for prog-mode buffers?
> >
> > What for?  No one complained about it yet, and leaving it on helps
> > find bugs and inefficiencies in the bidi display engine.  So my vote
> > is NAY.
> 
> Sounds like the plan is to make Emacs painful to use for everyone in
> order to make it a priority for L2R users to solve R2L problems.

If L2R users don't want bidirectional support any place near them, to
the degree that they are unwilling to help make it better at the cost
of some moderate effort, please tell so now.  I will then rip out
everything I did, and will forbid the FSF to ever use that code in any
version of Emacs they distribute.  That won't make up for the 2 years
of efforts I invested in making this happen, but at least I will cut
my losses and won't have to go through the ordeal of arguing with one
David Kastrup anymore.

Re: bidi-display-reordering is now non-nil by default

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>> Date: Tue, 16 Aug 2011 09:54:18 +0200
>> 
>> Andreas Schwab <schwab@linux-m68k.org> writes:
>> 
>> > Eli Zaretskii <eliz@gnu.org> writes:
>> >
>> >> Translated into Emacs-speak, this means we need a variable that, when
>> >> bound to some special value, instructs the reordering engine to treat
>> >> certain characters as segment separators.
>> >
>> > They already exist: comment-start and comment-end, for example.
>> 
>> I am having a somewhat hard time imagining it as a good thing if Emacs
>> displays source code that does no longer correspond to a reasonably
>> straightforward manner of printing the file.  reordering source into
>> something that "makes more sense" seems quite more invasive than mere
>> font-locking.
>
> Please explain what you mean by "no longer corresponds".  If a comment
> or a string in a source file use R2L scripts, how would you like Emacs
> to display them?

With the normal bidi algorithm.

> As for printing, don't modern printers and associated software already
> reorder text when they print?

According to the normal bidi algorithm, but hardly by recognizing and
treating comment fields on their own.

> If not, we may be able to use Emacs facilities to DTRT here.

If "the right thing" is different from the rest of the world, working
with the text is tied into Emacs.

>> I am not fond of the idea of writing code that gets unreadable unless
>> you read it back with Emacs, or even a specific version of Emacs or
>> specific settings.
>
> ??? The discussion was about _displaying_, no one suggested writing
> anything to the file.  The idea was to use _existing_ characters or
> strings that delimit the relevant portions of source as a cue for
> reordering those delimited parts in some way that makes them readable.

If the byte stream in the file represents an unreadable result, then
Emacs should better facilitate creating a byte stream representing a
readable result (for example, by semiautomatically inserting direction
marks when the result would otherwise start looking bad) instead of
presenting a more readable result that does not corresponding to the
file contents in a straightforward manner.

When used as a mere file viewer, there is something to be said for using
higher level information than the bidi algorithm in order to give a
rendition that is better than the original.

But if Emacs is used as an editor, a writing tool, it should help to
write readable source, and presenting a more readable letter order than
the bidi algorithm or a printer or a browser could come up with is
misleading the writer.

-- 
David Kastrup

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: cyd@stupidchicken.com,  dak@gnu.org,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Tue, 16 Aug 2011 11:10:33 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Andreas Schwab <schwab@linux-m68k.org>
> >> Cc: Chong Yidong <cyd@stupidchicken.com>,  dak@gnu.org,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> >> Date: Tue, 16 Aug 2011 09:40:45 +0200
> >> 
> >> Eli Zaretskii <eliz@gnu.org> writes:
> >> 
> >> > Translated into Emacs-speak, this means we need a variable that, when
> >> > bound to some special value, instructs the reordering engine to treat
> >> > certain characters as segment separators.
> >> 
> >> They already exist: comment-start and comment-end, for example.
> >
> > These are strings, not characters.  I meant single characters.
> 
> You can't have that.  The field separators are multi-character strings.

If all of them can be made to behave like segment separators, then
Chong's idea will still work.  If not, then it won't.

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Tue, 16 Aug 2011 11:40:28 +0200
> 
> If the byte stream in the file represents an unreadable result, then
> Emacs should better facilitate creating a byte stream representing a
> readable result (for example, by semiautomatically inserting direction
> marks when the result would otherwise start looking bad) instead of
> presenting a more readable result that does not corresponding to the
> file contents in a straightforward manner.

That's fine with me, but I thought people were saying that compilers
and interpreters might not like these directional controls.  If that's
not true, using them would be the most natural and simple solution,
IMO.

Does someone know how other programming IDEs treat R2L text in
comments in strings?

> But if Emacs is used as an editor, a writing tool, it should help to
> write readable source, and presenting a more readable letter order than
> the bidi algorithm or a printer or a browser could come up with is
> misleading the writer.

Assuming that other programming editors don't have similar features
that don't depend on directional controls, I agree.  Displaying text
in a way that is very different from other similar tools is not a good
idea.

Re: bidi-display-reordering is now non-nil by default

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>> Date: Tue, 16 Aug 2011 09:07:16 +0200
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> From: Chong Yidong <cyd@stupidchicken.com>
>> >
>> >> since there is no near-term solution, how bout turning off bidi
>> >> display reordering for prog-mode buffers?
>> >
>> > What for?  No one complained about it yet, and leaving it on helps
>> > find bugs and inefficiencies in the bidi display engine.  So my vote
>> > is NAY.
>> 
>> Sounds like the plan is to make Emacs painful to use for everyone in
>> order to make it a priority for L2R users to solve R2L problems.
>
> If L2R users don't want bidirectional support any place near them,

You are putting words into my mouth.

> to the degree that they are unwilling to help make it better at the
> cost of some moderate effort, please tell so now.

Most users are _unable_ to help.  The pretest versions are for mostly
intended for those who _are_ likely able to help.  The betas and the
final releases, however, are also for everybody else.

We have refrained from enabling features like font-locking by default in
releases until the costs of using them for everyone were under control.

Nobody is arguing to remove R2L support.  The question is about what
defaults are sensible to put into a release.  You have yourself worked
rather hard to make sure that one can have performance comparable to
before by disabling bidi support altogether.

At the current point of time, I don't think that anybody suggests
bidi-display-reordering be set to nil in the release version: work is
good enough that this need not be an issue.

Your standard reaction for regressions encountered in connection with
discussions about both performance problems as well as more fine-grained
defaults is threats.  That is not productive.  I understand that you
have invested a large amount of work into R2L typesetting, and I
understand that it is vital for your personal resources (including your
motivation) that others join in the effort.

> I will then rip out everything I did, and will forbid the FSF to ever
> use that code in any version of Emacs they distribute.  That won't
> make up for the 2 years of efforts I invested in making this happen,
> but at least I will cut my losses and won't have to go through the
> ordeal of arguing with one David Kastrup anymore.

Which is silly since I am not even a developer of Emacs anymore.  I
handed back my commit rights years ago after a similar discussion with
you.

I am now unsubscribing from the Emacs developer list in order to save
you having to worry about my person.

But you really should think whether your way of dealing with technical
problems by ostracizing is doing yourself a favor, let alone anybody
else.

-- 
David Kastrup

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Tue, 16 Aug 2011 12:01:22 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: David Kastrup <dak@gnu.org>
> >> Date: Tue, 16 Aug 2011 09:07:16 +0200
> >> 
> >> Eli Zaretskii <eliz@gnu.org> writes:
> >> 
> >> >> From: Chong Yidong <cyd@stupidchicken.com>
> >> >
> >> >> since there is no near-term solution, how bout turning off bidi
> >> >> display reordering for prog-mode buffers?
> >> >
> >> > What for?  No one complained about it yet, and leaving it on helps
> >> > find bugs and inefficiencies in the bidi display engine.  So my vote
> >> > is NAY.
> >> 
> >> Sounds like the plan is to make Emacs painful to use for everyone in
> >> order to make it a priority for L2R users to solve R2L problems.
> >
> > If L2R users don't want bidirectional support any place near them,
> 
> You are putting words into my mouth.

No, I'm reading what they say loud and clear.

> > to the degree that they are unwilling to help make it better at the
> > cost of some moderate effort, please tell so now.
> 
> Most users are _unable_ to help.

Wrong.  They help _a_lot_ by using the display features and reporting
bugs.  I'm grateful to all of them, because I could never find these
problems myself.

> We have refrained from enabling features like font-locking by default in
> releases until the costs of using them for everyone were under control.

Whether or not the bidi display is good enough to hit the street
should be an issue near the end of the pretest, which has not even
started yet.  So raising these claims now is unjustified and premature
at best.

> Nobody is arguing to remove R2L support.  The question is about what
> defaults are sensible to put into a release.  You have yourself worked
> rather hard to make sure that one can have performance comparable to
> before by disabling bidi support altogether.

The possibility to disable bidi support is a debugging aid, that's
all.  That is why it is not a user variable.  Don't read anything else
into that.

> Your standard reaction for regressions encountered in connection with
> discussions about both performance problems as well as more fine-grained
> defaults is threats.

Hogwash.  My standard reaction to that is fixing every regression that
is being reported.  Anyone who looks at the bug tracker will see that.
When I disagree with suggestions to change the defaults, I explain
why.  I would expect my opinions and explanations to have some
considerable weight, precisely _because_ I'm working on this alone,
and therefore know much more about these issues than anyone else here.

> I understand that you have invested a large amount of work into R2L
> typesetting, and I understand that it is vital for your personal
> resources (including your motivation) that others join in the
> effort.

I have no problems keeping doing this alone.  But I need to know that
I have the support of the Emacs community in doing so.  It is okay to
ask questions about the technical solutions that I implement or
suggest, or suggest alternatives.  But as soon as my _motives_ are
questioned, as soon as someone dares to suggest that I'm deliberately
putting users through suffering for some personal reasons, that is the
sign of lack of support.  I will not go on without support.  I refuse
to work on features that will not be used in a released Emacs, for no
good reason except that it brings more complexity and perhaps some
moderate slowdown.

If someone thinks that complexity and some slowdown can be avoided
while doing so much more, then they are dreaming.  Every significant
feature we put in Emacs always causes similar effects.  Testing them
in a variety of environments and workflows is the only way of getting
them improved and enhanced for the users.  This feature is no
different.

> But you really should think whether your way of dealing with technical
> problems by ostracizing is doing yourself a favor, let alone anybody
> else.

I have no problem with your technical comments, never had them.  What
triggered this exchange was a nasty personal attack on myself and my
motives, which had nothing to do with technicalities.  That I will not
tolerate, not from you, not from anyone else.

Re: bidi-display-reordering is now non-nil by default

[Stefan Monnier]

> "Higher level protocols" don't include futzing with bidirectional
> properties of characters.  "Higher level protocol" means some means to
> determine segment boundaries other than segment separator characters
> that are part of the text.  Translated into Emacs-speak, this means we
> need a variable that, when bound to some special value, instructs the
> reordering engine to treat certain characters as segment separators.

Maybe we then need a way to declare some chars (such as SPC,
punctuation, string markers, comment markers, ...) as
segment boundaries.

>> since there is no near-term solution, how bout turning off bidi
>> display reordering for prog-mode buffers?
> What for?  No one complained about it yet, and leaving it on helps
> find bugs and inefficiencies in the bidi display engine.  So my vote
> is NAY.

Agreed.


        Stefan

Re: bidi-display-reordering is now non-nil by default

[Stefan Monnier]

> Does someone know how other programming IDEs treat R2L text in
> comments in strings?

Indeed, I think we first need to be more clear about the problem with
which we're concerned.
AFAIK there are a few different cases:
- R2L in comments, strings, and doc.
- R2L in identifiers.
If we manage to mark comment/string/doc delimiters as
bidi-field-delimiters, then the first problem is solved.
But the second is still open: I wouldn't want

  (foo ARGA ARGB)

to be displayed as

  (foo (BGRA AGRA


        Stefan

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Chong Yidong <cyd@stupidchicken.com>,  dak@gnu.org,  emacs-devel@gnu.org
> Date: Tue, 16 Aug 2011 10:03:15 -0400
> 
> > "Higher level protocols" don't include futzing with bidirectional
> > properties of characters.  "Higher level protocol" means some means to
> > determine segment boundaries other than segment separator characters
> > that are part of the text.  Translated into Emacs-speak, this means we
> > need a variable that, when bound to some special value, instructs the
> > reordering engine to treat certain characters as segment separators.
> 
> Maybe we then need a way to declare some chars (such as SPC,
> punctuation, string markers, comment markers, ...) as
> segment boundaries.

As I wrote, implementing this for an arbitrary set of characters is
easy.  But the problem is that any character which we declare a
segment separator will behave like that everywhere in that buffer.  If
the same characters appear in a different context, where we do want
the reordering, we are screwed.

For example, suppose we declare SPC a segment separator.  Then this:

  (foo ARGA ARGB)

will be displayed correctly, but blanks in doc strings in the same
buffer will mess up the display, because something like this:

  "FOO BAR BAZ"

will be displayed like this:

  "OOF RAB ZAB"

whereas we want this:

  "ZAB RAB OOF"

Re: bidi-display-reordering is now non-nil by default

[Chong Yidong]

Eli Zaretskii <eliz@gnu.org> writes:

>> since there is no near-term solution, how bout turning off bidi
>> display reordering for prog-mode buffers?
>
> What for?  No one complained about it yet

I think I mentioned in an earlier message that the presence of RTL
characters mucks up the display of Emacs Lisp buffers.  This happend to
me while writing code containing RTL strings to test
string-mark-left-to-right.

> leaving it on helps find bugs and inefficiencies in the bidi display
> engine.

I understand your point, but it's not realistic to expect anyone to
"find bugs and inefficiencies" in source code buffers right now.  Those
buffers aren't displayed in an intelligible manner, because they require
missing "higher-level protocols" to display properly.

It makes a lot of sense for Emacs 24.1 to provide just the "bare" UAX9
algorithm, enabled by default in text modes (where bare UAX9 will likely
DTRT) and special modes (where Emacs can segment by hand if necessary);
with the higher-level segmentation functionality postphoned to the next
release.  In prog modes, which require that higher-level segmentation
functionality, it then makes sense to disable bidi display for now.

This would also give a lot more time to study different ways of
implementing segmentation (e.g. definining segmentation characters vs
extending the bidi code to recognize text properties).

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Chong Yidong <cyd@stupidchicken.com>
> Cc: monnier@iro.umontreal.ca, dak@gnu.org, emacs-devel@gnu.org
> Date: Tue, 16 Aug 2011 11:48:54 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> since there is no near-term solution, how bout turning off bidi
> >> display reordering for prog-mode buffers?
> >
> > What for?  No one complained about it yet
> 
> I think I mentioned in an earlier message that the presence of RTL
> characters mucks up the display of Emacs Lisp buffers.  This happend to
> me while writing code containing RTL strings to test
> string-mark-left-to-right.

Won't a newline and/or LRM/RLM (the latter inside the string) fix
that?  If not, please show me the relevant examples.

In general, all reordering information is tossed at every newline and
restarted anew for the next line.  So judicious placement of newlines
should do the trick in most cases.

> Those buffers aren't displayed in an intelligible manner, because
> they require missing "higher-level protocols" to display properly.

I think you exaggerate.  Both strings and comments are mostly
displayed correctly, and a few nasty surprises can be fixed by
inserting newlines and sometimes directional control characters.
Sure, there's place for improvement: Emacs should not force the user
to format the source in some specific manner, for it to display
correctly.  But for someone who actually uses R2L characters in
comments and strings, this is a definite improvement compared to Emacs
23, where they had to read them backwards.

> In prog modes, which require that higher-level segmentation
> functionality, it then makes sense to disable bidi display for now.

I disagree (see above), but I'll go with anything you and Stefan
decide.

> This would also give a lot more time to study different ways of
> implementing segmentation (e.g. definining segmentation characters vs
> extending the bidi code to recognize text properties).

I don't think we should feel pressed to resolve this before Emacs
24.1, since the problem is not as acute as it may seem at first
glance.  So we have that time anyway.

Re: bidi-display-reordering is now non-nil by default

[Chong Yidong]

Eli Zaretskii <eliz@gnu.org> writes:

> Won't a newline and/or LRM/RLM (the latter inside the string) fix
> that?  In general, all reordering information is tossed at every
> newline and restarted anew for the next line.  So judicious placement
> of newlines should do the trick in most cases.

Having to juggle newlines when editing source code is no good.

The ideal long-term solution, I think, is to make the bidi display code
aware of text properties, as several people have already suggested.
Then it should be easy to exploit font-lock to give reasonably correct
bidi segmentation, e.g. by treating font-lock-comment-face and
font-lock-string-face boundaries as bidi segmentation boundaries.

For now, I don't feel strongly about the idea of turning off bidi
display in prog modes.  But if we don't, we should at least document
some of the pitfalls discussed in this thread, and maybe state that
having RTL script in source code buffers is currently a bad idea.

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Chong Yidong <cyd@stupidchicken.com>
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> Date: Tue, 16 Aug 2011 18:24:24 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Won't a newline and/or LRM/RLM (the latter inside the string) fix
> > that?  In general, all reordering information is tossed at every
> > newline and restarted anew for the next line.  So judicious placement
> > of newlines should do the trick in most cases.
> 
> Having to juggle newlines when editing source code is no good.

It's an annoyance, I agree.  We should lift this limitation as soon as
we can.

> The ideal long-term solution, I think, is to make the bidi display code
> aware of text properties, as several people have already suggested.

I'm afraid making the reordering engine aware of all text properties
will considerably slow down redisplay, due to the need to check
character properties very frequently.  It also runs a high risk of
completely blending the reordering code with the display engine, which
will make them both very hard to maintain; currently, they are clearly
separated.

In addition, the current design of bidi.c does not support starting
and stopping reordering on arbitrary character positions; it only does
that at BEGV and ZV.  If possible, I'd like to avoid changing that.

The idea I currently favor is to use the `display' properties for this
purpose.  bidi.c already takes notice of those `display' properties
which replace the underlying text with something else, like an image
or another text.  The text covered by such properties is reordered as
a single long character, and if a property is a string, it is then
reordered separately by pushing the iterator object onto a stack and
restarting the iteration, including reordering, anew for the display
string.

So suppose we define a new display spec, something like

 'display '(bidi-reorder BASE-DIRECTION)

where BASE-DIRECTION, either left-to-right or right-to-left, is the
"paragraph direction" to be in effect while reordering the text
covered by the display spec.  Then we add code to the display engine
to implement this spec by pushing the iterator, reinitializing the
iteration for the span of this display property, then popping it when
it's done.  In effect, we create a "display string" out of a chunk of
buffer text, and then reorder it separately.

With this feature, a major mode can put such display properties on
whichever parts of buffer text it wants.  This will work regardless of
whether the buffer itself has bidi-display-reordering set to nil or t,
because the implementation of such display spec will force reordering
for the covered text regardless of the parent buffer.

> Then it should be easy to exploit font-lock to give reasonably correct
> bidi segmentation, e.g. by treating font-lock-comment-face and
> font-lock-string-face boundaries as bidi segmentation boundaries.

We should be very careful with reusing font-lock as basis for
reordering, because the user has too much knobs to control font-lock.
For example, few of the font-lock features speed up redisplay by
deferring fontification to a later time.  With font-lock, this just
displays text in the default face; with reordering, it will flush
incorrectly rendered text for a perceptible amount of time.  I'm not
sure it's a good idea.

There are also people who disable font-lock completely.

We could use font-lock faces, when they exist, as a cue for placing
the `display' properties described above, though.

> For now, I don't feel strongly about the idea of turning off bidi
> display in prog modes.  But if we don't, we should at least document
> some of the pitfalls discussed in this thread, and maybe state that
> having RTL script in source code buffers is currently a bad idea.

I'll add this to my todo and revisit that during the pretest.

Re: bidi-display-reordering is now non-nil by default

[Juri Linkov]

>  'display '(bidi-reorder BASE-DIRECTION)

I wonder is it possible to add a new property `dir', so you have
to check just two properties `display' and `dir' in bidi.c?

In HTML, the `dir' attribute specifies the base direction (ltr, rtl) of text:

  <p dir="ltr">text</p>
  <p dir="rtl">text</p>

Info: http://www.w3.org/International/tutorials/bidi-xhtml/

In Emacs it would be good to do the same with similar text properties:

  (propertize "text" 'dir 'ltr)
  (propertize "text" 'dir 'rtl)

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Juri Linkov <juri@jurta.org>
> Cc: Chong Yidong <cyd@stupidchicken.com>,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Wed, 17 Aug 2011 12:34:32 +0300
> 
> >  'display '(bidi-reorder BASE-DIRECTION)
> 
> I wonder is it possible to add a new property `dir', so you have
> to check just two properties `display' and `dir' in bidi.c?
> 
> In HTML, the `dir' attribute specifies the base direction (ltr, rtl) of text:
> 
>   <p dir="ltr">text</p>
>   <p dir="rtl">text</p>

Your `dir' is my BASE-DIRECTION.  That is, modes that support HTML
will put a display property with the above form of spec on "text", and
set the BASE-DIRECTION value according to dir=.  So my proposal should
be sufficient for rendering HTML and XML as well.

Re: bidi-display-reordering is now non-nil by default

[Lars Magne Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> And currently string-mark-left-to-right does the job just fine.

I'm catching up on mail, so I might be responding to something that has
already been decided otherwise, but since `string-mark-left-to-right'
still seems to exist in the sources, I'm guessing it's still the
preferred solution.  :-)

Looking at the code, it just appends "\x200e" to the end of r2l strings,
right?  This means that for Gnus to just bind `gnus-tmp-subject-or-nil'
to `(string-mark-left-to-right gnus-tmp-subject-or-nil)' won't work --
the subjects are liable to be chopped off by length restriction specs.
So this will probably have to be special-cased somewhere in the
format-spec machinery, which should be possible without wreaking too
much havoc.

But it's unlikely that this will work properly for people who have
already customised their `gnus-summary-format-line', since that may have
to change.  I'll see what I can come up with...

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: bidi-display-reordering is now non-nil by default

[Chong Yidong]

Eli Zaretskii <eliz@gnu.org> writes:

> I'm afraid making the reordering engine aware of all text properties
> will considerably slow down redisplay, due to the need to check
> character properties very frequently.  It also runs a high risk of
> completely blending the reordering code with the display engine, which
> will make them both very hard to maintain; currently, they are clearly
> separated.

No, the lookup would be done at the redisplay engine level, not the
reordering engine level: add a new entry in it_props[] for handling a
(say) `bidi-override' text property.  Emacs would process this during
the step in redisplay where it handles other properties (like faces and
invisibility), and record the information into the iterator.  The bidi
code would take it from there.

That should not be particularly performance intensive; we are already
doing an equal or greater amount of work handling things like face
properties.

>> Then it should be easy to exploit font-lock to give reasonably correct
>> bidi segmentation, e.g. by treating font-lock-comment-face and
>> font-lock-string-face boundaries as bidi segmentation boundaries.
>
> We should be very careful with reusing font-lock as basis for
> reordering, because the user has too much knobs to control font-lock.
> For example, few of the font-lock features speed up redisplay by
> deferring fontification to a later time.  With font-lock, this just
> displays text in the default face; with reordering, it will flush
> incorrectly rendered text for a perceptible amount of time.  I'm not
> sure it's a good idea.

The fundamental issue is that correctly segmenting source code requires
knowledge of the underlying syntax.  Sure, it's possible to come up with
some hacks that "mostly" work, but font lock is already there, so we
ought to try to use it first.

For this reason, I'm not about concerned about the deferred
fontification issue: if you want Emacs to segment properly, you'd want
it to do an amount of work equivalent to font-lock anyway.

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Chong Yidong <cyd@stupidchicken.com>
> Date: Wed, 17 Aug 2011 18:32:46 -0400
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I'm afraid making the reordering engine aware of all text properties
> > will considerably slow down redisplay, due to the need to check
> > character properties very frequently.  It also runs a high risk of
> > completely blending the reordering code with the display engine, which
> > will make them both very hard to maintain; currently, they are clearly
> > separated.
> 
> No, the lookup would be done at the redisplay engine level, not the
> reordering engine level: add a new entry in it_props[] for handling a
> (say) `bidi-override' text property.  Emacs would process this during
> the step in redisplay where it handles other properties (like faces and
> invisibility), and record the information into the iterator.  The bidi
> code would take it from there.

This won't work, not with the way the reordering engine is currently
integrated with redisplay.  The reason is that above the reordering
level, the iteration through buffer text is non-linear.  Your
suggestion assumes that the redisplay iterator will bump into this new
text property _before_ it processes the text which follows it.  But
this assumption is false because of the non-linear scanning of the
buffer text.

Let me show an example to illustrate how the bidirectional display
handles text properties.  Suppose you have the following buffer text
(as usual, capital letters mean R2L characters):

   000000111110000
   abcde ABCDE xyz
         ^    ^

The number above each character shows the text properties of the
characters; 0 means no properties, 1 means some specific property.
This example shows only one property, spanning only the R2L
characters; the real-life examples can be much more complex.  The '^'
characters below show the "stop positions" computed by the iterator --
those are the buffer positions where display engine should process the
text property by calling one or more handlers in the it_props[] array,
filling the iterator with attributes necessary for displaying the text
until the next "stop position".

To move from the blank character between `e' and `A' to the next
character in visual order, the display iterator calls the reordering
engine.  When it does that, the first (leftmost) "stop position" was
not yet acted upon, because the current iterator position is smaller
than that stop.  When the call to the reordering engine returns, it
sets the iterator position at `E', since the ABCDE part should be
displayed as EDCBA on the screen.  Oops! we just missed the "stop
position".  What happens next is the redisplay engine realizes that
the stop position was missed, so it scans back to find the last "stop
position" preceding `E' (since there could be other text properties or
overlays in-between), and then handles it using the handlers in
it_props[]; see handle_stop_backwards for how this is done.  Then it
can deliver `E' with the right attributes, and continue delivering all
the successive characters, until it crosses some "stop position"
again, either going forward or backward.

This is why it won't work to control reordering with text properties:
by the time the redisplay engine realizes that there's another text
property to apply, a crucial part of reordering has already happened.
The bidi_it structure that is part of the iterator already has all the
information about reordering of "ABCDE", having scanned it all inside
a single call to bidi_move_to_visually_next.  That scan entirely
ignores all text properties except one: the `display' property, and
then only if its value will cause the covered text to be replaced by
something else, like an image or a string.

It would be possible, of course, to have the handler of the
`bidi-override' property to toss all the reordering information,
reposition to before `A' and start anew.  But that's a terrible waste
of cycles, especially if the text covered by that property is not so
short.  The waste is not only in that we will have to throw away
information we already gathered at some cost, but also because
repositioning the iterator to an arbitrary place means we need to
restart the bidi iteration from the beginning of the line in order to
have the correct state of the bidi iterator needed to continue from
that place; see get_visually_first_element for the details.

> >> Then it should be easy to exploit font-lock to give reasonably correct
> >> bidi segmentation, e.g. by treating font-lock-comment-face and
> >> font-lock-string-face boundaries as bidi segmentation boundaries.
> >
> > We should be very careful with reusing font-lock as basis for
> > reordering, because the user has too much knobs to control font-lock.
> > For example, few of the font-lock features speed up redisplay by
> > deferring fontification to a later time.  With font-lock, this just
> > displays text in the default face; with reordering, it will flush
> > incorrectly rendered text for a perceptible amount of time.  I'm not
> > sure it's a good idea.
> 
> The fundamental issue is that correctly segmenting source code requires
> knowledge of the underlying syntax.  Sure, it's possible to come up with
> some hacks that "mostly" work, but font lock is already there, so we
> ought to try to use it first.

Font-lock just uses regexps and syntax tables.  Everything else in
font-lock is meant to avoid the annoyingly long delay it takes to
fully fontify a large buffer.  What I'm saying is that, apart from
using regexps and syntax tables, the considerations and trade-offs
that are valid for font-lock are not necessarily valid for
bidirectional display.

> For this reason, I'm not about concerned about the deferred
> fontification issue: if you want Emacs to segment properly, you'd want
> it to do an amount of work equivalent to font-lock anyway.

Amount of work is the least of my concerns in this regard.  I'm
worried about the effect the temporarily incorrect display will have
on users.

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Mon, 15 Aug 2011 14:13:21 -0400
> Cc: dak@gnu.org, emacs-devel@gnu.org
> 
> And currently string-mark-left-to-right does the job just fine.

In line with the above, does the following patch look like TRT?  It is
needed, e.g., when displaying completions for C-x C-f when some of the
files in the directory end in R2L characters.  Since we cover the TAB
with a `(display (space :align-to ...))' property, which makes it a
``replacing'' display property, the underlying TAB loses its
bidirectional properties and is reordered as if it were a neutral
character.  So instead of

  OOF                RAB

I see

  RAB                OOF

which shows the candidates in the wrong order, especially if
completions-format is `vertical'.

If this patch is okay, can you tell whether other completion
facilities in Emacs might need similar changes?

=== modified file 'lisp/minibuffer.el'
--- lisp/minibuffer.el	2011-08-15 16:10:39 +0000
+++ lisp/minibuffer.el	2011-08-18 13:07:52 +0000
@@ -1119,13 +1119,24 @@ It also eliminates runs of equal strings
 				       `(display (space :align-to ,column)))
 		  nil))))
             (if (not (consp str))
-                (put-text-property (point) (progn (insert str) (point))
+                (put-text-property (point)
+				   (progn
+				     (insert (bidi-string-mark-left-to-right str))
+				     (point))
                                    'mouse-face 'highlight)
-              (put-text-property (point) (progn (insert (car str)) (point))
+              (put-text-property (point)
+				 (progn
+				   (insert
+				    (bidi-string-mark-left-to-right (car str)))
+				   (point))
                                  'mouse-face 'highlight)
-              (add-text-properties (point) (progn (insert (cadr str)) (point))
+              (add-text-properties (point)
+				   (progn
+				     (insert
+				      (bidi-string-mark-left-to-right (cadr str)))
+				     (point))
                                    '(mouse-face nil
-                                     face completions-annotations)))
+						face completions-annotations)))
 	    (cond
 	     ((eq completions-format 'vertical)
 	      ;; Vertical format

Re: bidi-display-reordering is now non-nil by default

[Chong Yidong]

Eli Zaretskii <eliz@gnu.org> writes:

> What happens next is the redisplay engine realizes that the stop
> position was missed, so it scans back to find the last "stop position"
> preceding `E' (since there could be other text properties or overlays
> in-between), and then handles it using the handlers in it_props[]; see
> handle_stop_backwards for how this is done.  Then it can deliver `E'
> with the right attributes, and continue delivering all the successive
> characters, until it crosses some "stop position" again, either going
> forward or backward.

Thanks for the explanation; I think I understand the problem.  But if
this is a fundamental limitation, that would be extremely disappointing.
The use of HTML spans for bidi segmentation is (IIUC) fairly well-known,
and character properties are the most straightforward analog in Emacs.

Here's another rough idea: when handle_stop is computing the next stop
position, it could call another function (analogous to compute_stop_pos)
to find and record the boundary of the next bidi-segmenting character
property---i.e. a "bidi stop position".  Then the reordering function
could look at this to help figure out the segmentation breaks, thus
avoiding advancing past this position.  Would something like that work?

> Amount of work is the least of my concerns in this regard.  I'm
> worried about the effect the temporarily incorrect display will have
> on users.

I don't recall the last time I ran into a problem with deferred
font-lock.  I seriously doubt this is a real issue.

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Chong Yidong <cyd@stupidchicken.com>
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> Date: Thu, 18 Aug 2011 13:13:40 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > What happens next is the redisplay engine realizes that the stop
> > position was missed, so it scans back to find the last "stop position"
> > preceding `E' (since there could be other text properties or overlays
> > in-between), and then handles it using the handlers in it_props[]; see
> > handle_stop_backwards for how this is done.  Then it can deliver `E'
> > with the right attributes, and continue delivering all the successive
> > characters, until it crosses some "stop position" again, either going
> > forward or backward.
> 
> Thanks for the explanation; I think I understand the problem.  But if
> this is a fundamental limitation, that would be extremely disappointing.
> The use of HTML spans for bidi segmentation is (IIUC) fairly well-known,
> and character properties are the most straightforward analog in Emacs.

What's wrong with using the `display' property with a special value,
as I suggested a few mails ago?  It _is_ a character property, isn't
it?

> Here's another rough idea: when handle_stop is computing the next stop
> position, it could call another function (analogous to compute_stop_pos)
> to find and record the boundary of the next bidi-segmenting character
> property---i.e. a "bidi stop position".  Then the reordering function
> could look at this to help figure out the segmentation breaks, thus
> avoiding advancing past this position.  Would something like that work?

Yes.  But using the `display' property means fewer changes, because we
already look for it during bidi iteration, and because we do almost
the same as needed here to reorder display strings.

> > Amount of work is the least of my concerns in this regard.  I'm
> > worried about the effect the temporarily incorrect display will have
> > on users.
> 
> I don't recall the last time I ran into a problem with deferred
> font-lock.  I seriously doubt this is a real issue.

??? Set jit-lock-defer-time to something like 2, then visit xdisp.c,
scroll, and watch the fun.

It's not a "problem", deferred font-lock is _supposed_ to work like
that.  Except that what is tolerable for font-lock might not be for
display reordering.

Re: bidi-display-reordering is now non-nil by default

[Chong Yidong]

Eli Zaretskii <eliz@gnu.org> writes:

> What's wrong with using the `display' property with a special value,
> as I suggested a few mails ago?  It _is_ a character property, isn't
> it?

I'll have to think more about it, but I am not sure it's a good idea.
The display property is already used for a lot of other unrelated
purposes, like variable-width spaces.  So if you want to mark out a bidi
segment on a stretch of text that already has other display properties
present, you'd currently have to do some nasty merging of display specs.
Searching for the bidi display spec would be similarly annoying.  It
would be much nicer to have a separate text property specifically for
the purpose of marking out a bidi segment.

>> I don't recall the last time I ran into a problem with deferred
>> font-lock.  I seriously doubt this is a real issue.
>
> ??? Set jit-lock-defer-time to something like 2, then visit xdisp.c,
> scroll, and watch the fun.

Then don't do that.

Obviously, there is a trade-off: exploiting font-lock means more
"magically correct" segmenting; using a hack (like marking chars
special) means less dependence on the font-lock machinery.  I don't
think the latter is worth it.

Re: bidi-display-reordering is now non-nil by default

[Stefan Monnier]

> Obviously, there is a trade-off: exploiting font-lock means more
> "magically correct" segmenting; using a hack (like marking chars
> special) means less dependence on the font-lock machinery.  I don't
> think the latter is worth it.

The problem is not jit-lock-defer-time but the fact that font-lock may
be turned off.  We could do it in syntax-propertize, of course, but it's
not great either (e.g. syntax-propertize is applied to all the text
before window-start whether that text will ever be displayed or not).


        Stefan

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Chong Yidong <cyd@stupidchicken.com>
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> Date: Thu, 18 Aug 2011 18:44:44 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > What's wrong with using the `display' property with a special value,
> > as I suggested a few mails ago?  It _is_ a character property, isn't
> > it?
> 
> I'll have to think more about it, but I am not sure it's a good idea.
> The display property is already used for a lot of other unrelated
> purposes, like variable-width spaces.  So if you want to mark out a bidi
> segment on a stretch of text that already has other display properties
> present, you'd currently have to do some nasty merging of display specs.

I don't see any need for any merging.  We already process display
specs on before- and after-strings, and the same code can be used
here.  IOW, those other display specs will be simply processed as part
of displaying the stretch of text covered by the bidi display spec.
Since the iterator state is pushed onto the stack, I see no issues
here that need merging.  If I'm missing something, please point out
what that is or show an example where we would need to merge display
specs.

> Searching for the bidi display spec would be similarly annoying.

We already do search for display properties as part of bidi scanning,
see bidi_fetch_char and compute_display_string_pos.  It would be a
simple job to add another spec to the list of those considered
relevant by handle_display_spec, which compute_display_string_pos
calls to see if the display spec needs special handling by the bidi
iteration.  What's left is to write a next_element_from_buffer_segment
or some such that is almost an identical twin of
next_element_from_string, and a few lines of code that initialize the
bidi iterator for reordering the segment, again almost like what we
already have for display strings.

> It would be much nicer to have a separate text property specifically
> for the purpose of marking out a bidi segment.

We can do that, too, but searching for 2 different properties is more
expensive than searching for just one.

Also, it isn't clear to me how your suggestion will work if the buffer
whose segments are determined by the bidi property is also reordered
"normally", e.g. if the surrounding text includes R2L characters.  The
non-linear iteration through the surrounding text will complicate the
way we take notice of the segments that are to be reordered
separately.

> >> I don't recall the last time I ran into a problem with deferred
> >> font-lock.  I seriously doubt this is a real issue.
> >
> > ??? Set jit-lock-defer-time to something like 2, then visit xdisp.c,
> > scroll, and watch the fun.
> 
> Then don't do that.

Users currently customize font-lock for purposes that have nothing to
do with bidi reordering.  With your suggestion, they will have two
contradicting goals for a single customization.  I don't see how they
will succeed in reconciling them.

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
> Date: Thu, 18 Aug 2011 23:16:13 -0400
> 
> > Obviously, there is a trade-off: exploiting font-lock means more
> > "magically correct" segmenting; using a hack (like marking chars
> > special) means less dependence on the font-lock machinery.  I don't
> > think the latter is worth it.
> 
> The problem is not jit-lock-defer-time but the fact that font-lock may
> be turned off.

Right, I mentioned that as well.

> We could do it in syntax-propertize, of course, but it's not great
> either (e.g. syntax-propertize is applied to all the text before
> window-start whether that text will ever be displayed or not).

So are you saying that using text properties for this is not a good
idea at all?  If so, what other Emacs features do we have to mark
segments of text for "special" reordering?  Markers, perhaps?

I think we should also think how will we use this feature if it
existed, before we go out to implement the feature.  What specific
segments of text we would want to mark, and in what kinds of
buffers/strings?  What mechanisms can be used to compute where to put
these markers, and how these mechanisms will be triggered?  How
exactly to handle HTML/XML with such a feature?  Etc., etc. -- all of
these questions can have a profound effect on the details of the
implementation, or even show that the idea is stillborn.

Re: bidi-display-reordering is now non-nil by default

[Lars Magne Ingebrigtsen]

Chong Yidong <cyd@stupidchicken.com> writes:

> It would be much nicer to have a separate text property specifically
> for the purpose of marking out a bidi segment.

Obviously I know nothing about any of this, so I'm just talking as
someone who would like to use this stuff without knowing anything much
about what Emacs does internally.

To me, a nice interface to create a bit of text (that may later be
chopped up before insertion) would be to have a function like
(bidi-make-separate-string "lala").  This could perhaps be implemented
like (propertize "lala" 'bidi-region (make-unique-identifier)).

That is, what I'm saying is "make a string that will be treated
independently for bidi purposes".  If I insert two such strings next to
each other, I want them to be separate, even if they happen to have the
same l2r/r2l properties, so there would be no merging of regions.

I think.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Date: Fri, 19 Aug 2011 16:51:03 +0200
> 
> To me, a nice interface to create a bit of text (that may later be
> chopped up before insertion) would be to have a function like
> (bidi-make-separate-string "lala").  This could perhaps be implemented
> like (propertize "lala" 'bidi-region (make-unique-identifier)).
> 
> That is, what I'm saying is "make a string that will be treated
> independently for bidi purposes".

Is using the directional control characters and/or TABs permitted for
the solution?  If it is, there's more than one way to do what you
want, I will show you the most bulletproof one as soon as you agree to
see it ;-)

If using directional controls is not allowed, or frowned upon, then we
currently lack the infrastructure to do that, and as you see from this
thread, exactly which feature would be best for doing this is still
being debated.

Btw, I'm not at all sure the API you suggest is the best one.  One
alternative would be to specify the entire line, with marker
characters to show separate fields, and ask to modify this line such
that fields will be displayed left to right (or right to left) in
their order in the string.  It's possible there are other alternatives
as well.

Re: bidi-display-reordering is now non-nil by default

[Chong Yidong]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Obviously, there is a trade-off: exploiting font-lock means more
>> "magically correct" segmenting; using a hack (like marking chars
>> special) means less dependence on the font-lock machinery.  I don't
>> think the latter is worth it.
>
> The problem is not jit-lock-defer-time but the fact that font-lock may
> be turned off.  We could do it in syntax-propertize, of course, but it's
> not great either (e.g. syntax-propertize is applied to all the text
> before window-start whether that text will ever be displayed or not).

I don't think we have to be concerned with the set of users who (i) wish
to turn off font-lock and (ii) edit source code with RTL characters and
(iii) expect those RTL segments to be automagically segmented according
to the programming language syntax.  Relax any one of those assumptions
and things are fine.

Re: bidi-display-reordering is now non-nil by default

[Chong Yidong]

Eli Zaretskii <eliz@gnu.org> writes:

>> I'll have to think more about it, but I am not sure it's a good idea.
>> The display property is already used for a lot of other unrelated
>> purposes, like variable-width spaces.  So if you want to mark out a bidi
>> segment on a stretch of text that already has other display properties
>> present, you'd currently have to do some nasty merging of display specs.
>
> I don't see any need for any merging.  We already process display
> specs on before- and after-strings, and the same code can be used
> here.  IOW, those other display specs will be simply processed as part
> of displaying the stretch of text covered by the bidi display spec.

Suppose you have a stretch of text which includes characters with no
text properties plus a space with the display property

   '(space :width 3.0)

Suppose you want to use a display text property to mark that stretch of
text as a single bidi segment.  You'd add '(bidi-foo blahblah) to most
of the characters, but the property on the space character would have to
be something like

   '((bidi-foo blahblah) (space :width 3.0))

to avoid clobbering the pre-existing space display spec.

Obviously it is possible to do this, but it is just way cleaner and more
user-friendly to use a separate property for this purpose.

Re: bidi-display-reordering is now non-nil by default

[Chong Yidong]

Eli Zaretskii <eliz@gnu.org> writes:

> I think we should also think how will we use this feature if it
> existed, before we go out to implement the feature.  What specific
> segments of text we would want to mark, and in what kinds of
> buffers/strings?

Here is one example: in Emacs Lisp source code, I want every Lisp string
to be treated as a single bidi segment (paragraph).  Whichever way this
is done, these segments ought to correspond well (or exactly) to what
font-lock-mode marks with font-lock-string-face.  In particular, the
bidi segmentation should not be confused by stuff like escaped \" chars.

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Chong Yidong <cyd@stupidchicken.com>
> Cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> Date: Fri, 19 Aug 2011 15:43:38 -0400
> 
>    '((bidi-foo blahblah) (space :width 3.0))
> 
> to avoid clobbering the pre-existing space display spec.
> 
> Obviously it is possible to do this, but it is just way cleaner and more
> user-friendly to use a separate property for this purpose.

If that's what you meant by "merging", then I simply misunderstood,
sorry.

However, handle_display_spec already handles such Lisp data structures
in display properties, so it's quite possible only minimum or no
changes will be needed to support this.  The only inconvenience is for
the Lisp code that would need to be more complicated when it puts
these properties.

Having a separate property has a disadvantage of being more expensive,
so unless someone implements a way of searching for a specified set of
properties, not just one or any one, I would prefer not to do that.
Especially since the plausibility of having other display properties
on the R2L segments is quite low, IMO.

Re: bidi reordering in program source buffers (was: bidi-display-reordering is now non-nil by default)

[Eli Zaretskii]

I've started a new topic, and suggest to have a separate one for each
issue that was discussed under the old name.

> From: Chong Yidong <cyd@stupidchicken.com>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, emacs-devel@gnu.org
> Date: Fri, 19 Aug 2011 16:00:30 -0400
> 
> Here is one example: in Emacs Lisp source code, I want every Lisp string
> to be treated as a single bidi segment (paragraph).

The example of strings in program source was on the table since the
beginning of this discussion, so it's obvious it should be supported.
I actually meant to have a discussion about HTML/XML buffers.

But even strings have some aspects that we never considered in enough
detail.  What about this use case (from C-like programming languages,
but similar problems exist with `format' in Lisp)?

   printf ("ABCDEF %d\n", foo);

If all we do is reorder the string as a single R2L paragraph, we get
this jumble:

   printf ("d\n% FEDCBA", foo);

which is simply illegible.  And if we reorder it as a single L2R
paragraph, we get this:

  printf ("FEDCBA %d\n", foo);

which is a bit better, but still wrong.  The correct display, that
corresponds to how this will look on any bidi-aware display, would be
this:

  printf ("%d FEDCBA\n", foo);

Note that the correct rendering depends not only on the bidi types of
characters in ABCDEF, but also in the characters produced by the
format specifier.  %d tells us exactly what these characters would be,
but what if we had a %s specifier, which could be anything?

Evidently, just following the cues from font-lock is not nearly enough
to DTRT even in this very simple use case.  For example, even if we
want to decide whether to reorder as R2L or L2R paragraph, font-lock
gives no clues.

Also, Stefan mentioned Lisp symbols.  Do we want to support symbols
that use R2L characters?  (One use case where it could be necessary
are those Lisp packages that store arbitrary words as symbols.)  If we
do, just following font-lock will flood any Lisp buffer with text
properties, with the net effect of slowing redisplay, even if none of
the symbols use R2L characters, which will happen in the absolute
majority of use cases.  Again, following font-lock sounds like a
design that doesn't scale well, and punishes the absolute majority of
use cases for the benefit of a tiny fraction of them.  That doesn't
sound right to me.

Re: bidi reordering in program source buffers (was: bidi-display-reordering is now non-nil by default)

[Andreas Schwab]

Eli Zaretskii <eliz@gnu.org> writes:

> And if we reorder it as a single L2R paragraph, we get this:
>
>   printf ("FEDCBA %d\n", foo);
>
> which is a bit better, but still wrong.

IMHO this is the correct way to display it.  Consider when adding more
format directives, you must not reorder them so they still match the
order of arguments.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: bidi reordering in program source buffers (was: bidi-display-reordering is now non-nil by default)

[Eli Zaretskii]

> From: Andreas Schwab <schwab@linux-m68k.org>
> Cc: Chong Yidong <cyd@stupidchicken.com>,  monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Sat, 20 Aug 2011 11:28:23 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > And if we reorder it as a single L2R paragraph, we get this:
> >
> >   printf ("FEDCBA %d\n", foo);
> >
> > which is a bit better, but still wrong.
> 
> IMHO this is the correct way to display it.  Consider when adding more
> format directives, you must not reorder them so they still match the
> order of arguments.

Match the order on the screen or in the source file?  The latter is
not changed at all by reordering.  As for the former, any bidi-aware
format string should use directives with the parameter numbers
explicitly specified with the n$ parameter ordinals, so the user
looking at the result on the screen will know the correct order as a
side effect.

In general, having something like this on display:

 printf ("FEDCBA %d PONMLK %f ZYX %lu\n", foo, bar, baz);

for a source that in logical order reads

 printf ("ABCDEF %d KLMNOP %f XYZ %lu\n", foo, bar, baz);

which is what you suggest, is only marginally better than the original
logical-order string, because the reading order is ambiguous at best,
and downright undecipherable in more complex situations, where the
format string includes punctuation and L2R text.  So if we think the
above is acceptable, we might as well consider not reordering this at
all.

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

Ping!

> Date: Thu, 18 Aug 2011 19:14:38 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> > From: Stefan Monnier <monnier@iro.umontreal.ca>
> > Date: Mon, 15 Aug 2011 14:13:21 -0400
> > Cc: dak@gnu.org, emacs-devel@gnu.org
> > 
> > And currently string-mark-left-to-right does the job just fine.
> 
> In line with the above, does the following patch look like TRT?  It is
> needed, e.g., when displaying completions for C-x C-f when some of the
> files in the directory end in R2L characters.  Since we cover the TAB
> with a `(display (space :align-to ...))' property, which makes it a
> ``replacing'' display property, the underlying TAB loses its
> bidirectional properties and is reordered as if it were a neutral
> character.  So instead of
> 
>   OOF                RAB
> 
> I see
> 
>   RAB                OOF
> 
> which shows the candidates in the wrong order, especially if
> completions-format is `vertical'.
> 
> If this patch is okay, can you tell whether other completion
> facilities in Emacs might need similar changes?
> 
> === modified file 'lisp/minibuffer.el'
> --- lisp/minibuffer.el	2011-08-15 16:10:39 +0000
> +++ lisp/minibuffer.el	2011-08-18 13:07:52 +0000
> @@ -1119,13 +1119,24 @@ It also eliminates runs of equal strings
>  				       `(display (space :align-to ,column)))
>  		  nil))))
>              (if (not (consp str))
> -                (put-text-property (point) (progn (insert str) (point))
> +                (put-text-property (point)
> +				   (progn
> +				     (insert (bidi-string-mark-left-to-right str))
> +				     (point))
>                                     'mouse-face 'highlight)
> -              (put-text-property (point) (progn (insert (car str)) (point))
> +              (put-text-property (point)
> +				 (progn
> +				   (insert
> +				    (bidi-string-mark-left-to-right (car str)))
> +				   (point))
>                                   'mouse-face 'highlight)
> -              (add-text-properties (point) (progn (insert (cadr str)) (point))
> +              (add-text-properties (point)
> +				   (progn
> +				     (insert
> +				      (bidi-string-mark-left-to-right (cadr str)))
> +				     (point))
>                                     '(mouse-face nil
> -                                     face completions-annotations)))
> +						face completions-annotations)))
>  	    (cond
>  	     ((eq completions-format 'vertical)
>  	      ;; Vertical format
> 
> 
> 

Re: bidi-display-reordering is now non-nil by default

[Stefan Monnier]

>> files in the directory end in R2L characters.  Since we cover the TAB
>> with a `(display (space :align-to ...))' property, which makes it a
>> ``replacing'' display property, the underlying TAB loses its
>> bidirectional properties and is reordered as if it were a neutral
>> character.  So instead of

I guess this brings us back to "a way to mark some char as a field
separator, just like a TAB; and in this particular case it clearly would
be fine to do it via a `display' property.

Some might even argue that a (space :align-to ...) display property is
sufficiently similar to a TAB that such a property should be interpreted
similarly to a field separator by the bidi reordering code.

>> If this patch is okay,

Assuming it's not straightforward to change the C code to handle such
display properties (not simple enough for 24.1, or maybe we're not sure
it's actually a good idea to do it), then your patch looks like an
OK solution.

>> can you tell whether other completion facilities in Emacs might need
>> similar changes?

I'd tend to think that most/all other completion facilities should be
fixed by using the generic code rather than by fixing their code, so
they shouldn't need similar changes.


        Stefan

Re: bidi-display-reordering is now non-nil by default

[Stefan Monnier]

>> +                (put-text-property (point)
>> +				   (progn
>> +				     (insert (bidi-string-mark-left-to-right str))
>> +				     (point))
>> 'mouse-face 'highlight)

The mouse-face property is used to delimit the text, but the above code
ends up marking not just STR but STR+LRM, which is not desirable
(e.g. choose-completion will end up choosing STR+LRM rather than jus STR).


        Stefan

Re: bidi-display-reordering is now non-nil by default

[Štěpán Němec]

On Mon, 22 Aug 2011 21:37:30 +0200
Stefan Monnier wrote:

>>> +                (put-text-property (point)
>>> +				   (progn
>>> +				     (insert (bidi-string-mark-left-to-right str))
>>> +				     (point))
>>> 'mouse-face 'highlight)
>
> The mouse-face property is used to delimit the text, but the above code
> ends up marking not just STR but STR+LRM, which is not desirable
> (e.g. choose-completion will end up choosing STR+LRM rather than jus
> STR).

OTOH, making choose-completion know better than relying on mouse-face
would also have the side-effect of providing more favourable conditions
for fixing bug#8897. ;-)

-- 
Štěpán

Re: bidi-display-reordering is now non-nil by default

[Stefan Monnier]

> OTOH, making choose-completion know better than relying on mouse-face
> would also have the side-effect of providing more favourable conditions
> for fixing bug#8897. ;-)

OTOH bug#8897 is for a use that's pretty far from the normal use case of
the completion code, so I wouldn't make such a significant change just to
accomodate this use case.


        Stefan

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Mon, 22 Aug 2011 15:35:24 -0400
> 
> I guess this brings us back to "a way to mark some char as a field
> separator, just like a TAB; and in this particular case it clearly would
> be fine to do it via a `display' property.
> 
> Some might even argue that a (space :align-to ...) display property is
> sufficiently similar to a TAB that such a property should be interpreted
> similarly to a field separator by the bidi reordering code.

Only :align-to, or any other properties supported by the `space'
display spec?  If only the former, why only that?  Why not :width, for
example?

> Assuming it's not straightforward to change the C code to handle such
> display properties (not simple enough for 24.1, or maybe we're not sure
> it's actually a good idea to do it)

It wouldn't be hard to add this feature, if you think it's okay to do
that now, the feature freeze notwithstanding.  I would need the answer
to the question above, though.

> >> can you tell whether other completion facilities in Emacs might need
> >> similar changes?
> 
> I'd tend to think that most/all other completion facilities should be
> fixed by using the generic code rather than by fixing their code, so
> they shouldn't need similar changes.

"Generic code" meaning treating a `space' display spec as a segment
separator?  If that's not what you meant, I don't understand what
other generic solution would be possible.

In any case, please humor me by giving the list of completion packages
outside of minibuffer.el, as my knowledge of this area barely covers
the standard completion facilities.

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Mon, 22 Aug 2011 15:37:30 -0400
> 
> >> +                (put-text-property (point)
> >> +				   (progn
> >> +				     (insert (bidi-string-mark-left-to-right str))
> >> +				     (point))
> >> 'mouse-face 'highlight)
> 
> The mouse-face property is used to delimit the text, but the above code
> ends up marking not just STR but STR+LRM, which is not desirable
> (e.g. choose-completion will end up choosing STR+LRM rather than jus STR).

I couldn't imagine the mouse-face is used for anything besides the
highlight.  Sounds like a kludge to me.

Anyway, if we will go the bidi-string-mark-left-to-right route, it
would be easy to fix this minor problem by putting the mouse-face on
the string before running it through bidi-string-mark-left-to-right.

Re: bidi-display-reordering is now non-nil by default

[Štěpán Němec]

On Tue, 23 Aug 2011 03:13:23 +0200
Stefan Monnier wrote:

>> OTOH, making choose-completion know better than relying on mouse-face
>> would also have the side-effect of providing more favourable conditions
>> for fixing bug#8897. ;-)
>
> OTOH bug#8897 is for a use that's pretty far from the normal use case of
> the completion code, so I wouldn't make such a significant change just to
> accomodate this use case.

I think I've gathered as much from the fact that the bug is still
unfixed. Which is precisely why I used this precious opportunity to
point out that there might be more reasons. So now there's also the fact
that using the face for this seems like a kludge to Eli (and possibly
others? certainly me included) and gets in the way of other unrelated
code (or at least makes certain "obvious" solutions "non-obvious").

-- 
Štěpán

use of `mouse-face' to delimit text zones [was: bidi-display-reordering is now non-nil by default]

[Drew Adams]

The question is about whether `mouse-face' should be used only for highlighting
(write-only), or whether it can be good practice to also examine `mouse-face'
boundaries to determine text zones.

Eli> I couldn't imagine the mouse-face is used for anything besides the
Eli> highlight.  Sounds like a kludge to me.

And yet `mouse-face' is used here and there throughout Emacs to determine the
boundaries of certain regions of text. If it were used only for highlighting
then you would find only its addition as a text property (i.e., write-only),
almost never its use/testing (e.g., `get-text-property',
`next(-single)-char-property').

Grep the Lisp sources for `mouse-face', then check occurrences where we retrieve
that property to determine the boundaries and position of various text
"candidates" (e.g. look for `next(-single)-char-property'). We do this for
comint history, Dired, tmm, gnus articles, quail completions,... 

It is true that in many libraries we only set the property, without ever
examining/reading it. But in other libraries we do examine it to determine
boundaries. This is not something new or unusual.

> > OTOH bug#8897 is for a use that's pretty far from the 
> > normal use case of the completion code, so I wouldn't
> > make such a significant change just to accomodate this use case.
> 
> now there's also the fact that using the [mouse-face property]
> for this seems like a kludge to Eli (and possibly others?
> certainly me included) and gets in the way of other unrelated
> code (or at least makes certain "obvious" solutions "non-obvious").

FWIW, in Icicles, just as in the vanilla Emacs completion code, I make heavy use
of the fact that the `mouse-face' property indicates the boundaries of
completion candidates. And in the case of Icicles, completion candidates are
often more complex (various text properties, embedded newline chars etc.).

I grant that perhaps a different text property could be used for completion
candidate delimiting in *Completions*.

But I expect it would anyway more or less need to coincide with the `mouse-face'
limits (though I might be missing something - haven't really followed this
thread or Štěpán's bug report).

Candidate layout (vertical, horizontal), cycling, sorting, highlighting, moving
to a candidate in a given position, grabbing a clicked candidate, drag-selecting
multiple candidates, etc. all depend on properly picking up candidate
boundaries. Sometimes such operations can be a bit complex. Today, Icicles, like
vanilla Emacs, uses `mouse-face' for that.

So I guess I echo what Stefan M. said. Logically I guess a different text
property could be used to delimit candidates, but things might be a lot more
complex if two properties were used or if displayed-text limits did not
correspond to the other (new) limits. Not to mention the extra hairiness
required for code like Icicles that supports multiple Emacs versions.

And tomorrow you or someone else might make the same argument about any new
property we introduced to delimit candidates. Yes, `mouse-face' is doing double
duty currently: highlight + delimit. I'm not convinced that's a bad thing. I
don't see it as a kludge, in any case.

All this is just to say that if you're counting voices (Eli + Štěpán vs Stefan),
count me, a priori, as not keen on abandoning the use of `mouse-face' to delimit
candidates. IOW, +1 for Stefan's reluctance in this regard.

[Wrt bug #8897, it sounds from a quick reading like the problem is not so much
with the use of `mouse-face' to delimit completion candidates as it is with some
hard-coding (implicit use) of `mouse-face' within the button code. But you can
ignore this comment, as I have not really followed that thread.]

Re: use of `mouse-face' to delimit text zones [was: bidi-display-reordering is now non-nil by default]

[Eli Zaretskii]

> From: "Drew Adams" <drew.adams@oracle.com>
> Date: Tue, 23 Aug 2011 08:29:22 -0700
> Cc: 'Eli Zaretskii' <eliz@gnu.org>, emacs-devel@gnu.org
> 
> Grep the Lisp sources for `mouse-face', then check occurrences where we retrieve
> that property to determine the boundaries and position of various text
> "candidates" (e.g. look for `next(-single)-char-property'). We do this for
> comint history, Dired, tmm, gnus articles, quail completions,... 
> [...]
> FWIW, in Icicles, just as in the vanilla Emacs completion code, I make heavy use
> of the fact that the `mouse-face' property indicates the boundaries of
> completion candidates.

Heavy use of this doesn't yet make it right.  Good (read:
"non-kludgey") programming practices should not use for some feature
symbols whose name or documentation give no clue of such a use.
Should I count the number of bugs submitted by you in the name of this
same principle?

The documentation of mouse-face says:

  `mouse-face'
       This property is used instead of `face' when the mouse is on or
       near the character.  For this purpose, "near" means that all text
       between the character and where the mouse is have the same
       `mouse-face' property value.

That's it.  Not a word about potential use of that for doing anything
else besides displaying the text.

How is Joe R. Hacker supposed to know that adding invisible characters
to a portion of text covered by mouse-face will break completion??

Btw, mouse-face can also be put on overlay strings and display
strings; how can you get the underlying text then?

IT'S A KLUDGE!!!

> things might be a lot more complex if two properties were used or if
> displayed-text limits did not correspond to the other (new) limits.

Like what?

> And tomorrow you or someone else might make the same argument about any new
> property we introduced to delimit candidates.

If the face is called `completion-candidate', who would "make the same
argument"?

Re: bidi-display-reordering is now non-nil by default

[Stefan Monnier]

>> I guess this brings us back to "a way to mark some char as a field
>> separator, just like a TAB; and in this particular case it clearly would
>> be fine to do it via a `display' property.
>> Some might even argue that a (space :align-to ...) display property is
>> sufficiently similar to a TAB that such a property should be interpreted
>> similarly to a field separator by the bidi reordering code.
> Only :align-to, or any other properties supported by the `space'
> display spec?

Good question.

> If only the former, why only that?

At least the :align-to has a clear similarity to the TAB char.

> Why not :width, for example?

I wouldn't know.  That's the problem.  I guess it would be a heuristic
in any case, so we'd want to provide some way to control the behavior in
case the heuristic is wrong.

>> Assuming it's not straightforward to change the C code to handle such
>> display properties (not simple enough for 24.1, or maybe we're not sure
>> it's actually a good idea to do it)
> It wouldn't be hard to add this feature, if you think it's okay to do
> that now, the feature freeze notwithstanding.  I would need the answer
> to the question above, though.

Maybe all `space' display properties should be treated as field
separators, but with a new :not-a-bidi-field-separator property
that can override this default.

Not sure which way the default should go, since OT1H I tend to feel like
the default should be to treat such spaces as field separators, based on
their similarity to TAB and based on the current way they're used in
tabulated data, but OTOH maybe this is a reflection of the lack of R2L
support until now.

>> >> can you tell whether other completion facilities in Emacs might need
>> >> similar changes?
>> I'd tend to think that most/all other completion facilities should be
>> fixed by using the generic code rather than by fixing their code, so
>> they shouldn't need similar changes.
> "Generic code" meaning treating a `space' display spec as a segment
> separator?

No, "generic code" meaning the "built-in" completion code in
minibuffer.el, as opposed to alternative completion packages that may
re-implement all or part of the completion code (ido, iswitchb,
auto-complete, company-mode, ...).

> In any case, please humor me by giving the list of completion packages
> outside of minibuffer.el, as my knowledge of this area barely covers
> the standard completion facilities.

For Emacs-24, I've made an effort to try and reduce the amount of
alternative completion code (most of which was not like
ido/iswitchb/.. but more like lisp-mode or makefile-mode
re-implementing the simple completion code already implemented for the
minibuffer), so I guess that nowadays there's mostly just ido and iswitchb
bundled with Emacs plus company-mode in the GNU ELPA (hopefully soon
accompanied with auto-complete and completion-ui).


        Stefan

Re: bidi-display-reordering is now non-nil by default

[Stefan Monnier]

> I think I've gathered as much from the fact that the bug is still
> unfixed. Which is precisely why I used this precious opportunity to
> point out that there might be more reasons. So now there's also the fact
> that using the face for this seems like a kludge to Eli (and possibly
> others? certainly me included) and gets in the way of other unrelated
> code (or at least makes certain "obvious" solutions "non-obvious").

Actually, changing it so that another property is added to delimit the
text wouldn't solve your problem: the mouse-face property would
still also be added and would hence still interfere with your use.


        Stefan

Re: use of `mouse-face' to delimit text zones [was: bidi-display-reordering is now non-nil by default]

[Stefan Monnier]

> That's it.  Not a word about potential use of that for doing anything
> else besides displaying the text.

Nothing prevents packages from (ab)using it for various other things at
the same time, of course.

> How is Joe R. Hacker supposed to know that adding invisible characters
> to a portion of text covered by mouse-face will break completion??

We're talking about a property added *by the completion code* and *for
the completion code*.  I.e. an internal use.  Joe R. Hacker doesn't need
to know about it unless he actually hacks on the code that handles
minibuffer-completion-help.

> Btw, mouse-face can also be put on overlay strings and display
> strings; how can you get the underlying text then?

Again, it's an internal use: the completion code doesn't use overlay
strings and doesn't care about display strings: it puts the mouse-face
property where it needs it and nowhere else.  It's been that way
probably ever since the *Completions* code was enhanced with mouse-face
highlighting (not sure what it used before).

Anyway, this is all irrelevant to this thread (I was just pointing out
the need for a trivial tweak to your patch if we intend to use it).


        Stefan

Re: use of `mouse-face' to delimit text zones

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Drew Adams <drew.adams@oracle.com>,  stepnem@gmail.com,  emacs-devel@gnu.org
> Date: Tue, 23 Aug 2011 14:34:25 -0400
> 
> > How is Joe R. Hacker supposed to know that adding invisible characters
> > to a portion of text covered by mouse-face will break completion??
> 
> We're talking about a property added *by the completion code* and *for
> the completion code*.  I.e. an internal use.

Not entirely true: when the mouse hovers above those parts of the
*Completions* buffer, the candidate under the mouse gets highlighted.
By contrast, a face added for internal purposed would be invisible to
the user.

> Joe R. Hacker doesn't need to know about it unless he actually hacks
> on the code that handles minibuffer-completion-help.

Guess what? he just did.

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Tue, 23 Aug 2011 14:19:37 -0400
> 
> >> I guess this brings us back to "a way to mark some char as a field
> >> separator, just like a TAB; and in this particular case it clearly would
> >> be fine to do it via a `display' property.
> >> Some might even argue that a (space :align-to ...) display property is
> >> sufficiently similar to a TAB that such a property should be interpreted
> >> similarly to a field separator by the bidi reordering code.
> > Only :align-to, or any other properties supported by the `space'
> > display spec?
> 
> Good question.
> 
> > If only the former, why only that?
> 
> At least the :align-to has a clear similarity to the TAB char.

I think they all do.  And the implementation is the same: we produce
white space of the specified dimensions.

Btw, I've looked at all the users of (space :SOMETHING) spec in
Emacs.  Almost all of them use :align-to.  The rest all use :width,
and with a single exception (ruler-mode, which uses this on the
fringes, and so is irrelevant to this discussion), all of them use
this to separate and align fields.

> Maybe all `space' display properties should be treated as field
> separators, but with a new :not-a-bidi-field-separator property
> that can override this default.

If we want to keep this simple enough to get into 24.1, I suggest to
treat them all as segment separators, and not introduce any overriding
properties until someone comes with a convincing use case that
actually needs something like that.

> > In any case, please humor me by giving the list of completion packages
> > outside of minibuffer.el, as my knowledge of this area barely covers
> > the standard completion facilities.
> 
> For Emacs-24, I've made an effort to try and reduce the amount of
> alternative completion code (most of which was not like
> ido/iswitchb/.. but more like lisp-mode or makefile-mode
> re-implementing the simple completion code already implemented for the
> minibuffer), so I guess that nowadays there's mostly just ido and iswitchb
> bundled with Emacs plus company-mode in the GNU ELPA (hopefully soon
> accompanied with auto-complete and completion-ui).

Thanks, I will take a look.

Need your decision on the space spec.

Re: bidi-display-reordering is now non-nil by default

[Štěpán Němec]

On Tue, 23 Aug 2011 20:24:56 +0200
Stefan Monnier wrote:

>> I think I've gathered as much from the fact that the bug is still
>> unfixed. Which is precisely why I used this precious opportunity to
>> point out that there might be more reasons. So now there's also the fact
>> that using the face for this seems like a kludge to Eli (and possibly
>> others? certainly me included) and gets in the way of other unrelated
>> code (or at least makes certain "obvious" solutions "non-obvious").
>
> Actually, changing it so that another property is added to delimit the
> text wouldn't solve your problem: the mouse-face property would
> still also be added and would hence still interfere with your use.

I don't think so. As explained in the bug thread, the problem for me is
that custom text properties (specifically mouse-face in my case) added
to the completion annotations are clobbered. I don't care about the face
of the completion items (although I'd argue they should respect user
additions as well). The argument you raised against not clobbering the
annotations additions and thus allowing face like mouse-face for them
was that the completions code would get confused, because that property
is used to delimit completion items. This argument would disappear if
that was fixed and a separate (non-face) property was used instead.

-- 
Štěpán

Re: use of `mouse-face' to delimit text zones

[Štěpán Němec]

On Tue, 23 Aug 2011 20:45:11 +0200
Eli Zaretskii wrote:

>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Cc: Drew Adams <drew.adams@oracle.com>,  stepnem@gmail.com,  emacs-devel@gnu.org
>> Date: Tue, 23 Aug 2011 14:34:25 -0400
>> 
>> > How is Joe R. Hacker supposed to know that adding invisible characters
>> > to a portion of text covered by mouse-face will break completion??
>> 
>> We're talking about a property added *by the completion code* and *for
>> the completion code*.  I.e. an internal use.
>
> Not entirely true: when the mouse hovers above those parts of the
> *Completions* buffer, the candidate under the mouse gets highlighted.
> By contrast, a face added for internal purposed would be invisible to
> the user.

IMO it needn't (shouldn't?) even be a face at all, just a separate text
property.

-- 
Štěpán

Re: bidi-display-reordering is now non-nil by default

[Stefan Monnier]

> Btw, I've looked at all the users of (space :SOMETHING) spec in
> Emacs.  Almost all of them use :align-to.  The rest all use :width,
> and with a single exception (ruler-mode, which uses this on the
> fringes, and so is irrelevant to this discussion), all of them use
> this to separate and align fields.

Thanks for looking at it.  It confirms my impression.

>> Maybe all `space' display properties should be treated as field
>> separators, but with a new :not-a-bidi-field-separator property
>> that can override this default.
> If we want to keep this simple enough to get into 24.1, I suggest to
> treat them all as segment separators, and not introduce any overriding
> properties until someone comes with a convincing use case that
> actually needs something like that.

OK.

>> For Emacs-24, I've made an effort to try and reduce the amount of
>> alternative completion code (most of which was not like
>> ido/iswitchb/.. but more like lisp-mode or makefile-mode
>> re-implementing the simple completion code already implemented for the
>> minibuffer), so I guess that nowadays there's mostly just ido and iswitchb
>> bundled with Emacs plus company-mode in the GNU ELPA (hopefully soon
>> accompanied with auto-complete and completion-ui).

> Thanks, I will take a look.

iswitchb already uses display-completion-list when displaying
completions in *Completions*.  And ido.el never displays anything in
*Completions*, AFAICT.
But both of them (along with icomplete.el) show a list of completions
"inline" in the minibuffer as in

  Switch to buffer: a{data.tex,foo.ada,*scratch*}

so maybe the {,,,} list should use bidi-string-mark-left-to-right before
the comma.  BTW, I think we should make it clear that
bidi-string-mark-left-to-right only makes sense in text that gets
displayed in L2R paragraphs, so it's largely restricted to buffers that
should set bidi-paragraph-direction to `left-to-right'.

> Need your decision on the space spec.

Is the above enough, or are there still more open issues?


        Stefan

Re: use of `mouse-face' to delimit text zones

[Stefan Monnier]

>> > How is Joe R. Hacker supposed to know that adding invisible characters
>> > to a portion of text covered by mouse-face will break completion??
>> We're talking about a property added *by the completion code* and *for
>> the completion code*.  I.e. an internal use.
> Not entirely true: when the mouse hovers above those parts of the
> *Completions* buffer, the candidate under the mouse gets highlighted.
> By contrast, a face added for internal purposed would be invisible to
> the user.

The completion code wants to add a mouse-face property so it gets
highlighted when the mouse hovers above it, yes.  That's not an
internal detail.  The internal detail is "how does it figure out the
element boundaries": the fact that it uses the mouse-face property for
that is an internal detail.

>> Joe R. Hacker doesn't need to know about it unless he actually hacks
>> on the code that handles minibuffer-completion-help.
> Guess what? he just did.

Luckily he sent his code for review, so someone knowledgeable in the
relevant code pointed out the pitfall he was about to fall into.


        Stefan

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Tue, 23 Aug 2011 15:17:29 -0400
> 
> iswitchb already uses display-completion-list when displaying
> completions in *Completions*.  And ido.el never displays anything in
> *Completions*, AFAICT.
> But both of them (along with icomplete.el) show a list of completions
> "inline" in the minibuffer as in
> 
>   Switch to buffer: a{data.tex,foo.ada,*scratch*}
> 
> so maybe the {,,,} list should use bidi-string-mark-left-to-right before
> the comma.

I will look at how they do this.

> BTW, I think we should make it clear that
> bidi-string-mark-left-to-right only makes sense in text that gets
> displayed in L2R paragraphs, so it's largely restricted to buffers that
> should set bidi-paragraph-direction to `left-to-right'.

The ELisp manual already says that.

> > Need your decision on the space spec.
> 
> Is the above enough, or are there still more open issues?

If the above is a "go ahead" for 24.1, then no open issues left.

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Tue, 23 Aug 2011 15:17:29 -0400
> 
>   Switch to buffer: a{data.tex,foo.ada,*scratch*}
> 
> so maybe the {,,,} list should use bidi-string-mark-left-to-right before
> the comma.

They should probably run each completion candidate through
bidi-string-mark-left-to-right, yes.

I say "probably" because there's a separate but related issue with
buffer and file names that include R2L characters and end in weak or
neutral characters: those weak or neutrals will be displayed to the
left of the R2L characters, as in "DCBA!" (instead of the more
plausible "!DCBA"), and appending the LRM will not cure that.  So
perhaps we should have a more comprehensive plan for handling these
issues before we start sprinkling bidi-string-mark-left-to-right all
over the place.

It is also possible we should add a more general bidi-string-mark-dwim
function, which uses either LRM or RLM depending on the paragraph
direction, to avoid additional changes later, when the UI will be
localizable.

Re: bidi-display-reordering is now non-nil by default

[Stefan Monnier]

>> Switch to buffer: a{data.tex,foo.ada,*scratch*}
>> so maybe the {,,,} list should use bidi-string-mark-left-to-right before
>> the comma.
> They should probably run each completion candidate through
> bidi-string-mark-left-to-right, yes.

That was my impression as well.

> I say "probably" because there's a separate but related issue with
> buffer and file names that include R2L characters and end in weak or
> neutral characters: those weak or neutrals will be displayed to the
> left of the R2L characters, as in "DCBA!" (instead of the more
> plausible "!DCBA"), and appending the LRM will not cure that.

I'm a bit confused here: you say "to the left of the R2L characters",
but your example shows it to the right.

> So perhaps we should have a more comprehensive plan for handling these
> issues before we start sprinkling bidi-string-mark-left-to-right all
> over the place.

Sounds fine to me, yes.  I don't think there's any hurry to try and
address all these issues before 24.1.

This said, it seems the problem you're referring to is a lot more
general than to icomplete/ido/iswitchb: the same problem will show up
wherever the file/buffer is displayed, no?

> It is also possible we should add a more general bidi-string-mark-dwim
> function, which uses either LRM or RLM depending on the paragraph
> direction, to avoid additional changes later, when the UI will be
> localizable.

Yes, that's what I was getting at, basically I think that we almost
never want to add a LRM but instead we want to add some kind of
segment-separator.
E.g. I think that a buffer ABC<1> should be either displayed as CBA<1>
or <1>CBA depending on the surrounding text direction (which might be
different from the paragraph direction).

> > > Need your decision on the space spec.
> > Is the above enough, or are there still more open issues?
> If the above is a "go ahead" for 24.1, then no open issues left.

Yes, it's a "go ahead" for the "treat any `space' display property as
a segment separator".


        Stefan

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Wed, 24 Aug 2011 10:51:34 -0400
> 
> >> Switch to buffer: a{data.tex,foo.ada,*scratch*}
> >> so maybe the {,,,} list should use bidi-string-mark-left-to-right before
> >> the comma.
> > They should probably run each completion candidate through
> > bidi-string-mark-left-to-right, yes.
> 
> That was my impression as well.
> 
> > I say "probably" because there's a separate but related issue with
> > buffer and file names that include R2L characters and end in weak or
> > neutral characters: those weak or neutrals will be displayed to the
> > left of the R2L characters, as in "DCBA!" (instead of the more
> > plausible "!DCBA"), and appending the LRM will not cure that.
> 
> I'm a bit confused here: you say "to the left of the R2L characters",
> but your example shows it to the right.

Yes, I meant the other left ;-)

> This said, it seems the problem you're referring to is a lot more
> general than to icomplete/ido/iswitchb: the same problem will show up
> wherever the file/buffer is displayed, no?

Yes, but with buffer/file contents it's a different story.  Whoever
writes the buffer/file text can (and should) use the directional
controls or accept the consequences.

By contrast, what Emacs shows in the echo area, the minibuffer, and
the buffers it pops up, like *Completions*, is something Emacs itself
generates, and the user has no control on those contents.  So we
should have some general policy in place, and have supporting
infrastructure, that would allow a Lisp program to DTRT with names of
files, buffers, etc. that we show in these contexts.

For example, when we prompt for something and show the default value,
the presence of R2L characters in that default should not produce the
kind of jumbled display we sometimes show with the current codebase.
Here's a simple example:

 emacs -Q
 M-: (read-buffer "Buffer name: " "אבגדה") RET
 M-n

> > It is also possible we should add a more general bidi-string-mark-dwim
> > function, which uses either LRM or RLM depending on the paragraph
> > direction, to avoid additional changes later, when the UI will be
> > localizable.
> 
> Yes, that's what I was getting at, basically I think that we almost
> never want to add a LRM but instead we want to add some kind of
> segment-separator.

It turns out a segment separator (in the UAX#9 definition, i.e. a TAB)
does not always help.  Here, for example, it does not, because the
colon and the closing parenthesis interfere:

     Buffer name (default אבגד):	12345

Remove the colon and the closing paren, and everything is dandy:

     Buffer name (default אבגד	12345

This is a strange behavior, it caught me by surprise.  But this
behavior is correct, as confirmed by Yudit and by the UAX#9 Reference
Implementation, so it's not a bug.  It's what the UBA mandates.

It looks like the only solution that always works is to use the
Paragraph Separator character (u+2029).  That's because each paragraph
is reordered separately and independently.

> E.g. I think that a buffer ABC<1> should be either displayed as CBA<1>
> or <1>CBA depending on the surrounding text direction (which might be
> different from the paragraph direction).

Not sure what you mean here.  Do you mean reorder that as a single
atomic unit?  If not, what do you mean by "surrounding text
direction"?  The surrounding text is different for each character of
the "<1>" string, and what we do with it by default is precisely
display depending on the surrounding text direction.

> > > > Need your decision on the space spec.
> > > Is the above enough, or are there still more open issues?
> > If the above is a "go ahead" for 24.1, then no open issues left.
> 
> Yes, it's a "go ahead" for the "treat any `space' display property as
> a segment separator".

Well, in light of the above, it sounds like we need to treat such
properties as a paragraph separator, to get the effect we need.  I'll
take a stab at that.

Re: bidi-display-reordering is now non-nil by default

[Stefan Monnier]

>> > I say "probably" because there's a separate but related issue with
>> > buffer and file names that include R2L characters and end in weak or
>> > neutral characters: those weak or neutrals will be displayed to the
>> > left of the R2L characters, as in "DCBA!" (instead of the more
>> > plausible "!DCBA"), and appending the LRM will not cure that.
>> I'm a bit confused here: you say "to the left of the R2L characters",
>> but your example shows it to the right.
> Yes, I meant the other left ;-)
>> This said, it seems the problem you're referring to is a lot more
>> general than to icomplete/ido/iswitchb: the same problem will show up
>> wherever the file/buffer is displayed, no?
> Yes, but with buffer/file contents it's a different story.

I meant buffer/file *name*, sorry.
But I think I now understand the problem you're talking about:
ABCD! all on its own would be properly displayed as !DCBA, but if it is
followed by an LRM mark then it will turn into DCBA!.  So, yes, that's
a problem.  What about adding a TAB instead of an LRM?

>  emacs -Q
>  M-: (read-buffer "Buffer name: " "אבגדה") RET
>  M-n

Right.  This is one more example of a "field".  I think we'd want
a function like bidi-independent-string which takes a string STR and
returns a new string that displays just like STR does but with the added
twist that none of the surrounding text will ever get displayed in the
middle of STR and that text that comes after STR gets displayed "after"
and text that comes before gets displayed "before".
I'm not sure if that is possible or even meaningful, tho.

>> E.g. I think that a buffer ABC<1> should be either displayed as CBA<1>
>> or <1>CBA depending on the surrounding text direction (which might be
>> different from the paragraph direction).
> If not, what do you mean by "surrounding text direction"?

Whether this ABC<1> appears within an L2R chunk of text or an L2R chunk
of text.  I guess that "paragraph direction" is a good approximation.

>> Yes, it's a "go ahead" for the "treat any `space' display property as
>> a segment separator".
> Well, in light of the above, it sounds like we need to treat such
> properties as a paragraph separator, to get the effect we need.  I'll
> take a stab at that.

Thanks,


        Stefan

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Thu, 25 Aug 2011 00:38:51 -0400
> 
> But I think I now understand the problem you're talking about:
> ABCD! all on its own would be properly displayed as !DCBA, but if it is
> followed by an LRM mark then it will turn into DCBA!.  So, yes, that's
> a problem.  What about adding a TAB instead of an LRM?

Won't help, you need an RLM after the exclam.  Or enclose the entire
string in RLE..PDF, see below.

> >  emacs -Q
> >  M-: (read-buffer "Buffer name: " "אבגדה") RET
> >  M-n
> 
> Right.  This is one more example of a "field".  I think we'd want
> a function like bidi-independent-string which takes a string STR and
> returns a new string that displays just like STR does but with the added
> twist that none of the surrounding text will ever get displayed in the
> middle of STR and that text that comes after STR gets displayed "after"
> and text that comes before gets displayed "before".
> I'm not sure if that is possible or even meaningful, tho.

It's possible.  It's called "embedding".  Here (watch the directional
control characters around the parentheses):

   Buffer name ‪(default "אבגדה"):‬ אבגדרשת123

Should we provide bidi-embed-string-l2r/r2l functions?

> >> E.g. I think that a buffer ABC<1> should be either displayed as CBA<1>
> >> or <1>CBA depending on the surrounding text direction (which might be
> >> different from the paragraph direction).
> > If not, what do you mean by "surrounding text direction"?
> 
> Whether this ABC<1> appears within an L2R chunk of text or an L2R chunk
> of text.  I guess that "paragraph direction" is a good approximation.

Why not always display it is <1>CBA ?  That's the natural visual
order.

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Thu, 25 Aug 2011 00:38:51 -0400
> 
> >> Yes, it's a "go ahead" for the "treat any `space' display property as
> >> a segment separator".
> > Well, in light of the above, it sounds like we need to treat such
> > properties as a paragraph separator, to get the effect we need.  I'll
> > take a stab at that.
> 
> Thanks,

Done in revision 105562 on the trunk.  Please give it a try.  Initial
testing with the *Completions* buffer indicates that it does TRT.
I've also removed the calls to bidi-string-mark-left-to-right from
buff-menu.el, as they are no longer needed there.

Re: bidi reordering in program source buffers (was: bidi-display-reordering is now non-nil by default)

[Ehud Karni]

On Sat, 20 Aug 2011 13:53:29 Eli Zaretskii <eliz@gnu.org> wrote:
>
> In general, having something like this on display:
>
>  printf ("FEDCBA %d PONMLK %f ZYX %lu\n", foo, bar, baz);
>
> for a source that in logical order reads
>
>  printf ("ABCDEF %d KLMNOP %f XYZ %lu\n", foo, bar, baz);
>
> which is what you suggest, is only marginally better than the original
> logical-order string, because the reading order is ambiguous at best,
> and downright undecipherable in more complex situations, where the
> format string includes punctuation and L2R text.  So if we think the
> above is acceptable, we might as well consider not reordering this at
> all.

I agree.

The default setting of bidi reordering for this kind of buffer should
be nil. I think a (global) function that toggles the bidi reordering
ONLY for the local buffer should ease the work of Emacs users.

Ehud.


--
 Ehud Karni           Tel: +972-3-7966-561  /"\
 Mivtach - Simon      Fax: +972-3-7976-561  \ /  ASCII Ribbon Campaign
 Insurance agencies   (USA) voice mail and   X   Against   HTML   Mail
 http://www.mvs.co.il  FAX:  1-815-5509341  / \
 GnuPG: 98EA398D <http://www.keyserver.net/>    Better Safe Than Sorry

Re: bidi reordering in program source buffers

[Eli Zaretskii]

> Date: Thu, 25 Aug 2011 16:51:54 +0300
> From: "Ehud Karni" <ehud@unix.mvs.co.il>
> Cc: schwab@linux-m68k.org, cyd@stupidchicken.com, monnier@iro.umontreal.ca,
>         emacs-devel@gnu.org
> 
> On Sat, 20 Aug 2011 13:53:29 Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > In general, having something like this on display:
> >
> >  printf ("FEDCBA %d PONMLK %f ZYX %lu\n", foo, bar, baz);
> >
> > for a source that in logical order reads
> >
> >  printf ("ABCDEF %d KLMNOP %f XYZ %lu\n", foo, bar, baz);
> >
> > which is what you suggest, is only marginally better than the original
> > logical-order string, because the reading order is ambiguous at best,
> > and downright undecipherable in more complex situations, where the
> > format string includes punctuation and L2R text.  So if we think the
> > above is acceptable, we might as well consider not reordering this at
> > all.
> 
> I agree.
> 
> The default setting of bidi reordering for this kind of buffer should
> be nil.

Thanks for the feedback.

So in your opinion, buffers that display source code should not be
reordered, not even the strings and comments they display?  IOW, we
should give up showing human-readable portions of those buffers in a
legible display form?  That's a bit harsh, no?

> I think a (global) function that toggles the bidi reordering
> ONLY for the local buffer should ease the work of Emacs users.

Can you elaborate why this is needed?

In any case, the function should simply toggle the value of
bidi-display-reordering, as it's per-buffer variable.  But I rather
think Emacs should set this variable correctly without user
intervention.

Re: bidi reordering in program source buffers

[Ehud Karni]

On Thu, 25 Aug 2011 20:28:30 Eli Zaretskii wrote:
>
> So in your opinion, buffers that display source code should not be
> reordered, not even the strings and comments they display?  IOW, we
> should give up showing human-readable portions of those buffers in a
> legible display form?  That's a bit harsh, no?

Yes, not even in comments or strings, see more below.

> > I think a (global) function that toggles the bidi reordering
> > ONLY for the local buffer should ease the work of Emacs users.
>
> Can you elaborate why this is needed?
>
> In any case, the function should simply toggle the value of
> bidi-display-reordering, as it's per-buffer variable.  But I rather
> think Emacs should set this variable correctly without user
> intervention.

I meant a command like `toggle-truncate-lines' or `toggle-word-wrap'.
Then the user can assign it to a key, so she can easily change from
logical to bidi ordered. In this way she can easily edit the R2L text
when she needs, but with one keystroke see the logic of the program
even if the R2L text is harder to read.

On second thought, may be the command should change the input method
too ?

Ehud.


--
 Ehud Karni           Tel: +972-3-7966-561  /"\
 Mivtach - Simon      Fax: +972-3-7976-561  \ /  ASCII Ribbon Campaign
 Insurance agencies   (USA) voice mail and   X   Against   HTML   Mail
 http://www.mvs.co.il  FAX:  1-815-5509341  / \
 GnuPG: 98EA398D <http://www.keyserver.net/>    Better Safe Than Sorry

Re: bidi reordering in program source buffers

[Eli Zaretskii]

> Date: Thu, 25 Aug 2011 23:01:38 +0300
> From: "Ehud Karni" <ehud@unix.mvs.co.il>
> Cc: cyd@stupidchicken.com, schwab@linux-m68k.org, monnier@iro.umontreal.ca,
>         emacs-devel@gnu.org
> 
> On Thu, 25 Aug 2011 20:28:30 Eli Zaretskii wrote:
> >
> > So in your opinion, buffers that display source code should not be
> > reordered, not even the strings and comments they display?  IOW, we
> > should give up showing human-readable portions of those buffers in a
> > legible display form?  That's a bit harsh, no?
> 
> Yes, not even in comments or strings, see more below.
> 
> > > I think a (global) function that toggles the bidi reordering
> > > ONLY for the local buffer should ease the work of Emacs users.
> >
> > Can you elaborate why this is needed?
> >
> > In any case, the function should simply toggle the value of
> > bidi-display-reordering, as it's per-buffer variable.  But I rather
> > think Emacs should set this variable correctly without user
> > intervention.
> 
> I meant a command like `toggle-truncate-lines' or `toggle-word-wrap'.
> Then the user can assign it to a key, so she can easily change from
> logical to bidi ordered. In this way she can easily edit the R2L text
> when she needs, but with one keystroke see the logic of the program
> even if the R2L text is harder to read.

But without any reasonable way of displaying strings in printf-like
function calls, what good will it do to the user to turn on
reordering?  She will just see jumbled text, with no real way of
knowing the reading order except moving the cursor with C-f.

OTOH, if we do have a reasonable way of showing the reordered text,
then why not have it on at all times?

> On second thought, may be the command should change the input method
> too ?

Strings are not necessarily made of only R2L characters.  Only the
user knows which parts she wants to edit.

Re: bidi-display-reordering is now non-nil by default

[Stefan Monnier]

> It's possible.  It's called "embedding".  Here (watch the directional
> control characters around the parentheses):
>    Buffer name ‪(default "אבגדה"):‬ אבגדרשת123
> Should we provide bidi-embed-string-l2r/r2l functions?

That sounds like just what we need, then.  How 'bout providing
a (bidi-embed-string STR &optional DIRECTION) where DIRECTION works like
bidi-paragraph-direction (i.e. nil means auto-detect).

>> Whether this ABC<1> appears within an L2R chunk of text or an L2R chunk
>> of text.  I guess that "paragraph direction" is a good approximation.
> Why not always display it is <1>CBA ?  That's the natural visual order.

To me, in an L2R context, any buffer name should be displayed as
NAME<N>, regardless of whether NAME happens to be R2L or L2R (i.e. NAME
should be an "embedded-string").

> Done in revision 105562 on the trunk.  Please give it a try.  Initial
> testing with the *Completions* buffer indicates that it does TRT.
> I've also removed the calls to bidi-string-mark-left-to-right from
> buff-menu.el, as they are no longer needed there.

Great, thanks,


        Stefan

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Thu, 25 Aug 2011 23:55:42 -0400
> 
> > It's possible.  It's called "embedding".  Here (watch the directional
> > control characters around the parentheses):
> >    Buffer name ‪(default "אבגדה"):‬ אבגדרשת123
> > Should we provide bidi-embed-string-l2r/r2l functions?
> 
> That sounds like just what we need, then.  How 'bout providing
> a (bidi-embed-string STR &optional DIRECTION) where DIRECTION works like
> bidi-paragraph-direction (i.e. nil means auto-detect).

Auto-detect is not really well-defined in this case, because the
DWIM-ish value of DIRECTION is determined by 3 factors:

 . the direction of the paragraph where STR will be inserted
 . the text around the position where it will be inserted
 . the desired effect of embedding the string

As you see, nothing here depends on the string itself.  The first 2
factors could be perhaps taken care of by providing additional
arguments that tell where the string will be inserted, although in
many cases the surrounding text will not be known yet (think inserting
a prompt into the minibuffer), at least not on both ends.  But the 3rd
factor is a complete mystery at this point, as I don't yet have enough
experience with using embeddings to enumerate all the possible use
cases.  If someone else can help produce such a list, I'm all ears.
Failing that, this function will have to be used by "experts" or after
some trial and error, and DIRECTION will have to be non-optional.

Alternatively, we could for now support nil as DIRECTION by assuming
the string will be inserted in the current buffer at point and using
the paragraph direction there.  It seems to DTRT in the simple use
cases I tried.  We could also add an optional BUFFER argument to use
the information from that buffer.

I guess what I'm saying is that the API will almost certainly change
as we gain experience.  So I'm unsure how elaborate we should make it
at this point, because any mistake will mean more changes in the
future when the API changes.

> >> Whether this ABC<1> appears within an L2R chunk of text or an L2R chunk
> >> of text.  I guess that "paragraph direction" is a good approximation.
> > Why not always display it is <1>CBA ?  That's the natural visual order.
> 
> To me, in an L2R context, any buffer name should be displayed as
> NAME<N>, regardless of whether NAME happens to be R2L or L2R (i.e. NAME
> should be an "embedded-string").

We will get what you want if we embed NAME in LRE..PDF when it's in an
L2R paragraph, and in RLE..PDF when it's in an R2L paragraph.  Or if
we insert LRM/RLM between NAME and <n>.  The difficulty is that you
need to know in advance the paragraph direction where the string will
be inserted.  This means the embedding should happen very close to
actually inserting the string.  IOW, utility functions such as
buffer-name cannot magically return a correctly decorated buffer name
and, what's worse, application code will need to take NAME<n> apart,
embed NAME or add the mark to it, then catenate <n> to it -- perhaps
we should have a function to do that.

There's a similar issue with file names: /abcd/ABCD/XYZ will be
displayed as /abcd/ZYX/DCBA.

Btw, using LRM/RLM is always preferred when possible, because it's
stateless, as opposed to an embedding.  UAX#9 says that much.  Also,
more single-byte encodings support the marks than the rest of the
directional controls.

So a related question is: should we prefer the marks where possible,
or should we standardize on embeddings?  I'm not sure we are equipped
with enough experience to decide at this time.  However, an important
practical question is: which alternative to use to fix read_minibuf,
for example?

Maybe we should embed fields and use the marks elsewhere?

Re: bidi-display-reordering is now non-nil by default

[Stefan Monnier]

> I guess what I'm saying is that the API will almost certainly change
> as we gain experience.  So I'm unsure how elaborate we should make it
> at this point, because any mistake will mean more changes in the
> future when the API changes.

Right.  Better not solve this problem yet.

> we insert LRM/RLM between NAME and <n>.  The difficulty is that you
> need to know in advance the paragraph direction where the string will
> be inserted.  This means the embedding should happen very close to

Hmm... so UAX#9 does not give us the tools we need, here.

> There's a similar issue with file names: /abcd/ABCD/XYZ will be
> displayed as /abcd/ZYX/DCBA.

IIUC in an English paragraph this should be displayed as /abcd/DCBA/ZYX
but in a Hebrew paragraph it should be displayed as ZYX/DCBA/abcd/, right?

> Btw, using LRM/RLM is always preferred when possible, because it's
> stateless, as opposed to an embedding.  UAX#9 says that much.  Also,

Yes, that makes a lot of sense (just think of using substring to select
part of the embedding, dropping the initial or final marker).

So embeddings should not be used too much, and/or should only be used
dynamically (jit-lock-style) and implicitly (e.g. not copied by
kill-region).

> So a related question is: should we prefer the marks where possible,
> or should we standardize on embeddings?  I'm not sure we are equipped
> with enough experience to decide at this time.

Agreed.

> However, an important practical question is: which alternative to use
> to fix read_minibuf, for example?

I don't think it matters too much; just use whatever works best.


        Stefan

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: emacs-devel@gnu.org
> Date: Fri, 26 Aug 2011 22:53:36 -0400
> 
> > There's a similar issue with file names: /abcd/ABCD/XYZ will be
> > displayed as /abcd/ZYX/DCBA.
> 
> IIUC in an English paragraph this should be displayed as /abcd/DCBA/ZYX
> but in a Hebrew paragraph it should be displayed as ZYX/DCBA/abcd/, right?

I think it should be displayed as /abcd/DCBA/ZYX in both cases.  When
a file name is mentioned in a R2L paragraph, it will need to be
embedded in LRE..PDF, as I did with the URL near the end of TUTORIAL.he.

> So embeddings should not be used too much, and/or should only be used
> dynamically (jit-lock-style) and implicitly (e.g. not copied by
> kill-region).

It depends.  If only part of the embedded text is copied, then the
directional formats should be dropped, indeed.  But if all of it is
copied, they should be kept, to preserve the visual appearance.

So I guess we will need to set filter-buffer-substring-functions to
something non-nil by default?  Or maybe modify filter-buffer-substring
itself to remove these characters as needed?

Re: bidi-display-reordering is now non-nil by default

[Stefan Monnier]

>> So embeddings should not be used too much, and/or should only be used
>> dynamically (jit-lock-style) and implicitly (e.g. not copied by
>> kill-region).
> It depends.  If only part of the embedded text is copied, then the
> directional formats should be dropped, indeed.  But if all of it is
> copied, they should be kept, to preserve the visual appearance.

Sounds difficult to implement reliably.

> So I guess we will need to set filter-buffer-substring-functions to
> something non-nil by default?  Or maybe modify filter-buffer-substring
> itself to remove these characters as needed?

Maybe that can be made to work, but we'd need to add a corresponding
filter-substring, and I'm not sure that we wouldn't get into bugs where
we need to change the code from buffer-substring to
filter-buffer-substring.


        Stefan

Re: bidi-display-reordering is now non-nil by default

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: emacs-devel@gnu.org
> Date: Sat, 27 Aug 2011 22:52:00 -0400
> 
> >> So embeddings should not be used too much, and/or should only be used
> >> dynamically (jit-lock-style) and implicitly (e.g. not copied by
> >> kill-region).
> > It depends.  If only part of the embedded text is copied, then the
> > directional formats should be dropped, indeed.  But if all of it is
> > copied, they should be kept, to preserve the visual appearance.
> 
> Sounds difficult to implement reliably.

What difficulties do you have in mind?

> > So I guess we will need to set filter-buffer-substring-functions to
> > something non-nil by default?  Or maybe modify filter-buffer-substring
> > itself to remove these characters as needed?
> 
> Maybe that can be made to work, but we'd need to add a corresponding
> filter-substring

What for? strings aren't copied by C-w and friends.

> and I'm not sure that we wouldn't get into bugs where we need to
> change the code from buffer-substring to filter-buffer-substring.

Which bugs, and why will we get into them?

Re: bidi-display-reordering is now non-nil by default

[Stefan Monnier]

>> >> So embeddings should not be used too much, and/or should only be used
>> >> dynamically (jit-lock-style) and implicitly (e.g. not copied by
>> >> kill-region).
>> > It depends.  If only part of the embedded text is copied, then the
>> > directional formats should be dropped, indeed.  But if all of it is
>> > copied, they should be kept, to preserve the visual appearance.
>> Sounds difficult to implement reliably.
> What difficulties do you have in mind?

See below.

>> > So I guess we will need to set filter-buffer-substring-functions to
>> > something non-nil by default?  Or maybe modify filter-buffer-substring
>> > itself to remove these characters as needed?
>> Maybe that can be made to work, but we'd need to add a corresponding
>> filter-substring
> What for? strings aren't copied by C-w and friends.

Most substrings are created non-interactively, many of them via
buffer-substring but others via just substring (sometimes/often on the
result of buffer-subtring).

>> and I'm not sure that we wouldn't get into bugs where we need to
>> change the code from buffer-substring to filter-buffer-substring.
> Which bugs, and why will we get into them?

Just think of the "Foo (default bar): " prompts, where `bar' often comes
from some buffer via buffer-substring/match-string.


        Stefan