bug#9571: 24.0.50; user option to turn off bidi, please

bug#9571: 24.0.50; user option to turn off bidi, please

[Drew Adams]

Dunno if this is still the latest word, but in the emacs-devel thread
"`C-b' is backward-char, `left' is left-char - why?", I was told that
the way for a user to turn off bidi is to set `bidi-display-reordering'
to nil.  Since it is buffer-local, that presumably means putting this
in .emacs if you want to turn it off everywhere:
 
 (setq-default bidi-display-reordering nil)
 
1. That's not very user-friendly.  We should have a user option that
does this, i.e., gives users an easy way to disable this feature if they
don't want to use it.
 
2. I see nothing in the Emacs doc that tells users clearly that if they
want to turn off bidi editing then they should set the default value of
this internal variable to nil.  Users should be told this.
 
The only thing said in the Emacs manual about this variable is this:
 
"The buffer-local variable `bidi-display-reordering' controls whether
text in the buffer is reordered for display.  If its value is
non-`nil', Emacs reorders characters that have right-to-left
directionality when they are displayed.  The default value is `t'."
 
That tells users who are inclined to study it how it works, but we
should be telling all users clearly how they can easily turn it off
everywhere, if they want to.

FWIW, I turn it off because both (a) it slows down Emacs (no, I don't
have a test case and I won't be coming up with one) and (b) I have no
need for bidi editing, at least for now.  If you don't believe (a), then
at least accept (b): I don't _want_ to use it.
 
In GNU Emacs 24.0.50.1 (i386-mingw-nt5.1.2600)
 of 2011-09-19 on 3249CTO
Windowing system distributor `Microsoft Corp.', version 5.1.2600
configured using `configure --with-gcc (4.5) --no-opt'
 

bug#9571: 24.0.50; user option to turn off bidi, please

[Eli Zaretskii]

tags 9571 + wontfix

> From: "Drew Adams" <drew.adams@oracle.com>
> Date: Wed, 21 Sep 2011 21:18:46 -0700
> 
> Dunno if this is still the latest word, but in the emacs-devel thread
> "`C-b' is backward-char, `left' is left-char - why?", I was told that
> the way for a user to turn off bidi is to set `bidi-display-reordering'
> to nil.  Since it is buffer-local, that presumably means putting this
> in .emacs if you want to turn it off everywhere:
>  
>  (setq-default bidi-display-reordering nil)

Yes.

> 1. That's not very user-friendly.  We should have a user option that
> does this, i.e., gives users an easy way to disable this feature if they
> don't want to use it.

No, at least not for now.

> 2. I see nothing in the Emacs doc that tells users clearly that if they
> want to turn off bidi editing then they should set the default value of
> this internal variable to nil.  Users should be told this.

No.  Consistent with the above.

> FWIW, I turn it off because both (a) it slows down Emacs (no, I don't
> have a test case and I won't be coming up with one) and (b) I have no
> need for bidi editing, at least for now.  If you don't believe (a), then
> at least accept (b): I don't _want_ to use it.

Suit yourself, but I see no need to make the bidi display a mode that
can be turned on and off.  So at least for Emacs 24.1 I'm not gonna
fix this.

bug#9571: 24.0.50; user option to turn off bidi, please

[Stefan Monnier]

> 1. That's not very user-friendly.  We should have a user option that
> does this, i.e., gives users an easy way to disable this feature if they
> don't want to use it.

Why would we need an option for that other than for debugging purposes?


        Stefan

bug#9571: 24.0.50; user option to turn off bidi, please

[Drew Adams]

> > 1. That's not very user-friendly.  We should have a user option that
> >    does this, i.e., gives users an easy way to disable this 
> >    feature if they don't want to use it.
> 
> Why would we need an option for that other than for debugging 
> purposes?

Uh, to give "users an easy way to disable this feature if they don't want to use
it."

Can you imagine that there are some Emacs users who do not _ever_ need or want
to edit bidirectional text?  Just like there are some users who never need or
want to use SQL code editing features, or image-library features, or Emacs-based
mail, or...

Why make such users figure out that they need to put a `setq-default' in their
.emacs to get back (as much as possible) the traditional behavior (and
possibly/hopefully eliminate some of the bidi overhead)?

I understand that it is apparently hard to make bidi support entirely optional,
e.g., remove all of the overhead even if turned off.  But it has also been
advised that _if_ you want to turn bidi editing off, setting this variable to
nil is the way to do it.

So make it a user option.  Only those users who want to turn it off will turn it
off - what's the problem?  Why not give users that control and ease of control?

bug#9571: 24.0.50; user option to turn off bidi, please

[Richard Stallman]

One use for a feature to disable bidi display
is to figure out what's really present in some strange piece of text.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use free telephony http://directory.fsf.org/category/tel/

bug#9571: 24.0.50; user option to turn off bidi, please

[Stefan Monnier]

> Can you imagine that there are some Emacs users who do not _ever_ need
> or want to edit bidirectional text?

If those users get a different behavior depending on
bidi-display-reordering, then we have a bug.


        Stefan

bug#9571: 24.0.50; user option to turn off bidi, please

[Stefan Monnier]

> One use for a feature to disable bidi display
> is to figure out what's really present in some strange piece of text.

"Figure out what's really present" is sufficiently vague that I don't
know what is the right way to address this need.  find-file-literally is
at least as likely to be effective, and in many more cases.


        Stefan

bug#9571: 24.0.50; user option to turn off bidi, please

[Štěpán Němec]

On Fri, 23 Sep 2011 06:11:16 +0200
Stefan Monnier wrote:

>> Can you imagine that there are some Emacs users who do not _ever_ need
>> or want to edit bidirectional text?
>
> If those users get a different behavior depending on
> bidi-display-reordering, then we have a bug.

I assume you don't consider UI responsiveness (buffer movement etc.)
deterioration a bug, then? AFIAK it's totally unrealistic to expect the
same responsiveness from Emacs with bidi enabled as from one with bidi
disabled, just as is the case with font locking. Given that font locking
already makes Emacs (next-to-?)unusable under certain circumstances, I'm
not sure easily discoverable and advertised possibility to switch bidi
off is such a very strange request from someone who never, ever, works
with bidi text (intentionally or not, i.e., doesn't even receive bidi
emails or something).

OTOH, I understand you probably just want to push bidi down everybody's
throat as much as possible to catch all problems this major change will
bring. So I think I understand the reluctance to honour Drew's request,
but the argumentation so far seems a bit disingenuous to me.

-- 
Štěpán

bug#9571: 24.0.50; user option to turn off bidi, please

[Eli Zaretskii]

> Date: Thu, 22 Sep 2011 17:44:39 -0400
> From: Richard Stallman <rms@gnu.org>
> Cc: 9571@debbugs.gnu.org
> 
> One use for a feature to disable bidi display
> is to figure out what's really present in some strange piece of text.

Bidi display does not change the characters on display or in the
buffer in any way.  So it will neither help nor prevent you to figure
out what's in the text.

As Stefan pointed out, find-file-literally has the effect of
automatically disabling bidi display (because the bidi reordering is
only done in multibyte buffers), so the same Emacs feature generally
used for figuring out what's in some strange text also inhibits the
bidi display, without any need for the user to do anything.

bug#9571: 24.0.50; user option to turn off bidi, please

[Juanma Barranquero]

2011/9/23 Štěpán Němec <stepnem@gmail.com>:

> I assume you don't consider UI responsiveness (buffer movement etc.)
> deterioration a bug, then? AFIAK it's totally unrealistic to expect the
> same responsiveness from Emacs with bidi enabled as from one with bidi
> disabled, just as is the case with font locking. Given that font locking
> already makes Emacs (next-to-?)unusable under certain circumstances, I'm
> not sure easily discoverable and advertised possibility to switch bidi
> off is such a very strange request from someone who never, ever, works
> with bidi text (intentionally or not, i.e., doesn't even receive bidi
> emails or something).

OOH, there was a time when suggesting that the user adds something to
.emacs, like

  (setq-default bidi-display-reordering nil)

was not considered obscure. Not everything must go through the
customization interface.

OTOH, you could equally argue that the new font backend stuff from the
Unicode merge slows Emacs for people who only ever edits ASCII, and
that disabling these backends should be an "easily disoverable and
advertised posibility". But currently the only way to disable a
specific font backend is via (initial|default)-frame-alist, and no one
has complained yet.

    Juanma

bug#9571: 24.0.50; user option to turn off bidi, please

[Eli Zaretskii]

> From: Štěpán Němec
> 	<stepnem@gmail.com>
> Date: Fri, 23 Sep 2011 10:01:08 +0200
> Cc: 9571@debbugs.gnu.org
> 
> On Fri, 23 Sep 2011 06:11:16 +0200
> Stefan Monnier wrote:
> 
> >> Can you imagine that there are some Emacs users who do not _ever_ need
> >> or want to edit bidirectional text?
> >
> > If those users get a different behavior depending on
> > bidi-display-reordering, then we have a bug.
> 
> I assume you don't consider UI responsiveness (buffer movement etc.)
> deterioration a bug, then?

Yes, I do.  Please report them as much as possible, so they could be
debugged and fixed.

> AFIAK it's totally unrealistic to expect the same responsiveness
> from Emacs with bidi enabled as from one with bidi disabled

No, it's not unrealistic.  In a vast majority of cases, the difference
in responsiveness is below the threshold of human perception.  If it
were not for that, we would have much more complaints about the new
display.  Where it exceeds that threshold, i.e. it becomes annoyingly
slow (whatever "annoyingly" means for you), please report that as a
bug with a clear test case.  Without reporting such incidents, there's
no way we can get the new display better as fast as needed, which I
hope is what everyone here wants.

> just as is the case with font locking.

This analogy is invalid, because font-lock is a very different beast.
For starters, it works on the entire buffer, while bidi is
display-only feature, and thus operates only on the (small) portion of
the buffer being displayed.

> OTOH, I understand you probably just want to push bidi down everybody's
> throat

No one around here has such a nasty attitude.  I'm appalled that you
can even suggest such a possibility.  The bidi display was designed
and implemented to be fast enough to be tolerable by people even if
they don't use any R2L scripts.  Where the reality flies at the face
of this design or implementation, there's a bug that needs to be
fixed.  But it's impossible to fix them if they are not reported,
because Emacs display features are a million, and no single living
person can envision every use case in a lifetime.

> the argumentation so far seems a bit disingenuous to me.

They are the truth nonetheless.  The old unidirectional display was
left in place only as a debugging aid.  Like any other basic feature
of the display engine, it will one day become the only way to display
text in Emacs, because maintaining two separate code bases, and in the
display engine at that, is a serious maintenance burden.  Introducing
a user-level variable to use the unidirectional display will be a
tremendous obstacle when we would like to get rid of the old code, I
hope at least that much is clear and agreed.

bug#9571: 24.0.50; user option to turn off bidi, please

[Štěpán Němec]

On Fri, 23 Sep 2011 11:21:02 +0200
Juanma Barranquero wrote:

> OOH, there was a time when suggesting that the user adds something to
> .emacs, like
>
>   (setq-default bidi-display-reordering nil)
>
> was not considered obscure. Not everything must go through the
> customization interface.

It's certainly good enough to me, but you won't find that advice in the
current documentation either, AFAICT.

> OTOH, you could equally argue that the new font backend stuff from the
> Unicode merge slows Emacs for people who only ever edits ASCII, and
> that disabling these backends should be an "easily disoverable and
> advertised posibility". But currently the only way to disable a
> specific font backend is via (initial|default)-frame-alist, and no one
> has complained yet.

Sure. If someone complained, it would probably idicate there was an
issue to be solved? As it happens, Drew did complain.

Also, this is not (only) about making `bidi-display-reordering' a custom
variable. It is about releasing Emacs 24 with still-not-widely-tested
bidi support, which quite possibly adds zero value for most users[1],
but does add significant problems at least in some (possibly as yet
unknown) circumstances. It might be a good idea, but I don't see why you
shouldn't be sincere about it and document the implications and
safeguard measures properly. IMO just saying something to the effect of
"bidi support can bring various problems and can be disabled by doing
AB, but we hope you don't disable it and instead help us improve it by
reporting any problems you encounter" might have better effect than
everybody switching off bidi for good after encountering the first
problem and then finding "Emacs 24 bidi slows things down, just turn it
off with AB" on the Internet.   

[1] Just in case: that of course doesn't mean it's not great or
necessary to have it.

-- 
Štěpán

bug#9571: 24.0.50; user option to turn off bidi, please

[Eli Zaretskii]

> From: Štěpán Němec
> 	<stepnem@gmail.com>
> Date: Fri, 23 Sep 2011 12:39:19 +0200
> Cc: 9571@debbugs.gnu.org
> 
> On Fri, 23 Sep 2011 11:21:02 +0200
> Juanma Barranquero wrote:
> 
> > OOH, there was a time when suggesting that the user adds something to
> > .emacs, like
> >
> >   (setq-default bidi-display-reordering nil)
> >
> > was not considered obscure. Not everything must go through the
> > customization interface.
> 
> It's certainly good enough to me, but you won't find that advice in the
> current documentation either, AFAICT.

??? The variable and its effect are clearly documented in both the
user manual and the ELisp manual.  Which is more than I can say for
the other new features introduced in Emacs 24, because documentation
is generally finalized only during the pretest.  What else besides
documentation of the variable is needed to make it clear to users how
to turn a feature off?

> As it happens, Drew did complain.

His complaint is not helpful, because he explicitly declined to
provide any details about the situation where bidi slowed down Emacs
for him.

> Also, this is not (only) about making `bidi-display-reordering' a custom
> variable. It is about releasing Emacs 24 with still-not-widely-tested
> bidi support, which quite possibly adds zero value for most users[1],
> but does add significant problems at least in some (possibly as yet
> unknown) circumstances.

We are not releasing Emacs 24.1 yet.  We didn't yet enter the pretest,
and it's anyone's guess when the pretest will be over.  Pretest is the
time when such radical new features are supposed to be tested and any
problems in them fixed before Emacs 24.1 hits the street.  We had an
even more radical change in the display engine when Emacs 21.1 was
developed; the pretest took more than a year back then.

bug#9571: 24.0.50; user option to turn off bidi, please

[Štěpán Němec]

On Fri, 23 Sep 2011 12:18:25 +0200
Eli Zaretskii wrote:

Thank you very much for the explanation. (I guess "pushing bidi down
everybody's throat" was picturesque in a wrong way, sorry for that; it's
really just another way of saying "bidi will (one day) become the only
way to display text in Emacs").

I still think "there is no way back" is all the more reason to be as
helpful and open about the changes and possible problems as possible. By
under-documenting the issue you only provide more opportunity to FUD. In
other words, not clearly saying that bidi might bring problems and how
to work around them in the user documentation at this point would only
be reasonable if Emacs bidi was pretty much a solved problem, but that's
really not the impression I got from the related emacs-devel discussions
or even from your explanation.

-- 
Štěpán

bug#9571: 24.0.50; user option to turn off bidi, please

[Juanma Barranquero]

2011/9/23 Štěpán Němec <stepnem@gmail.com>:

> Thank you very much for the explanation. (I guess "pushing bidi down
> everybody's throat" was picturesque in a wrong way, sorry for that; it's
> really just another way of saying "bidi will (one day) become the only
> way to display text in Emacs").

Again: the unicode merge brought the new font backends, which became
the only way to display text in Emacs. That's progress. It's not as if
the previous releases suddenly disappear, for those unable or
unwilling to use newer Emacsen. I switched from Firefox to Chrome
because I cannot bear the new Firefox interface, but I don't expect
the Firefox developers to add ways to make Firefox 6 to behave exactly
like Firefox 4.

> I still think "there is no way back" is all the more reason to be as
> helpful and open about the changes and possible problems as possible.

Pretest *is* the moment to discover, and hopefully fix, the possible problems...

> In
> other words, not clearly saying that bidi might bring problems and how
> to work around them in the user documentation at this point would only
> be reasonable if Emacs bidi was pretty much a solved problem, but that's
> really not the impression I got from the related emacs-devel discussions
> or even from your explanation.

That's a bit absurd, IMO (no offense intended). New features are never
a "solved problem" until they have been extensively tested, and then
some. That has to be done at some point in time. And that moment is
now.

    Juanma

bug#9571: 24.0.50; user option to turn off bidi, please

[Eli Zaretskii]

> From: Štěpán Němec <stepnem@gmail.com>
> Cc: 9571@debbugs.gnu.org
> Date: Fri, 23 Sep 2011 13:09:49 +0200
> 
> By under-documenting the issue you only provide more opportunity to
> FUD. In other words, not clearly saying that bidi might bring
> problems and how to work around them in the user documentation at
> this point would only be reasonable if Emacs bidi was pretty much a
> solved problem, but that's really not the impression I got from the
> related emacs-devel discussions or even from your explanation.

I happen to be a documentation freak, and tried to do my best in
documenting everything related to bidi.  If you can suggest specific
changes to the user manual or NEWS about this, please do.  Saying in
the manual that some feature "means trouble" is not something we use
to do, because users will say "if it's trouble, why don't you fix
it?".  We need to find a way of saying something useful that will help
users nonetheless.  Suggestions are welcome.

bug#9571: 24.0.50; user option to turn off bidi, please

[Richard Stallman]

    "Figure out what's really present" is sufficiently vague that I don't
    know what is the right way to address this need.  find-file-literally is
    at least as likely to be effective, and in many more cases.

find-file-literally disables character set decoding.  That means you'd
have to manually figure out how the bytes correspond to the characters
of the file, or else give a command to decode it.

If the confusion has to do with bidi, the feature you want for
understanding it is to turn off bidi and nothing else.  It has to be
easy to do, so why NOT do it?

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use free telephony http://directory.fsf.org/category/tel/

bug#9571: 24.0.50; user option to turn off bidi, please

[Richard Stallman]

    Bidi display does not change the characters on display or in the
    buffer in any way.  So it will neither help nor prevent you to figure
    out what's in the text.

If the reordering has confusing effects, turning off bidi display
would enable you to see clearly what the real order is in the buffer.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use free telephony http://directory.fsf.org/category/tel/

bug#9571: 24.0.50; user option to turn off bidi, please

[Štěpán Němec]

On Fri, 23 Sep 2011 13:45:35 +0200
Juanma Barranquero wrote:

> 2011/9/23 Štěpán Němec <stepnem@gmail.com>:

>> I still think "there is no way back" is all the more reason to be as
>> helpful and open about the changes and possible problems as possible.
>
> Pretest *is* the moment to discover, and hopefully fix, the possible problems...

It's meant to be, yes, but I suspect most users just wait for the
release. And in any case, clear information about an experimental (or
call it "new", if "experimental" is controversial) feature and related
issues is very useful for pretesters, too.

>> In
>> other words, not clearly saying that bidi might bring problems and how
>> to work around them in the user documentation at this point would only
>> be reasonable if Emacs bidi was pretty much a solved problem, but that's
>> really not the impression I got from the related emacs-devel discussions
>> or even from your explanation.
>
> That's a bit absurd, IMO (no offense intended). New features are never
> a "solved problem" until they have been extensively tested, and then
> some. That has to be done at some point in time. And that moment is
> now.

Some features are more experimental than others, some are more annoying
when gone wrong than others, and some cause annoyances harder to
diagnose than others.

On Fri, 23 Sep 2011 13:50:59 +0200
Eli Zaretskii wrote:

>> From: Štěpán Němec <stepnem@gmail.com>
>> Cc: 9571@debbugs.gnu.org
>> Date: Fri, 23 Sep 2011 13:09:49 +0200
>> 
>> By under-documenting the issue you only provide more opportunity to
>> FUD. In other words, not clearly saying that bidi might bring
>> problems and how to work around them in the user documentation at
>> this point would only be reasonable if Emacs bidi was pretty much a
>> solved problem, but that's really not the impression I got from the
>> related emacs-devel discussions or even from your explanation.
>
> I happen to be a documentation freak, and tried to do my best in
> documenting everything related to bidi.  If you can suggest specific
> changes to the user manual or NEWS about this, please do.  Saying in
> the manual that some feature "means trouble" is not something we use
> to do, because users will say "if it's trouble, why don't you fix
> it?".  We need to find a way of saying something useful that will help
> users nonetheless.  Suggestions are welcome.

When you take one example Lars reported recently -- a buffer where the
current bidi code wasn't able to diagnose the paragraph direction
properly (or something) in a relatively untypical buffer (a lot of long
lines without paragraph breaks). It made the buffer unusable. Now, how
would a user diagnose or solve this issue? How would the current
documentation help him in doing so?

The user has to figure out that 1) (a problem with) the bidi reordering
is the cause of the problem 2) they can work around it by setting
bidi-display-reordering to nil. I'm rather sure the current
documentation provides no clue as to point 1) (because it doesn't say
such problems might occur and there is no other way to know), and I'm
not quite sure it provides sufficient clues for point 2) (it says the
variable "controls whether text in the buffer is reordered for display";
is a user with no clue about bidi likely to understand that as "when set
to nil, the text in the buffer should behave just as before the bidi
feature was introduced"? I don't know, but I suspect I might not, if I
hadn't seen previous comments and references to the variable and its
effects on emacs-devel and other places).

I think unless you're pretty sure such problems are not likely to occur
again, you should at least say (in NEWS if not in the manual; I would
probably check the NEWS first in such situation anyway) that such things
can happen and how to work around it (and you should probably also say
that bidi is here to stay anyway so disabling it is only a workaround,
encouraging users to report issues instead of just turning bidi off).

-- 
Štěpán

bug#9571: 24.0.50; user option to turn off bidi, please

[Eli Zaretskii]

> Date: Fri, 23 Sep 2011 08:31:41 -0400
> From: Richard Stallman <rms@gnu.org>
> CC: drew.adams@oracle.com, 9571@debbugs.gnu.org
> 
>     Bidi display does not change the characters on display or in the
>     buffer in any way.  So it will neither help nor prevent you to figure
>     out what's in the text.
> 
> If the reordering has confusing effects, turning off bidi display
> would enable you to see clearly what the real order is in the buffer.

As I said (following Stefan), find-file-literally has that effect.

bug#9571: 24.0.50; user option to turn off bidi, please

[Eli Zaretskii]

> Date: Fri, 23 Sep 2011 08:31:33 -0400
> From: Richard Stallman <rms@gnu.org>
> Cc: 9571@debbugs.gnu.org
> 
> If the confusion has to do with bidi, the feature you want for
> understanding it is to turn off bidi and nothing else.

What confusion are you referring to?  Are we talking about a buffer
with or without R2L characters?  And how would setting
bidi-display-reordering to nil resolve that confusion?

IOW, please describe the relevant use cases in more detail.  Without
that, I fear we are having a misunderstanding of some kind.

> It has to be easy to do, so why NOT do it?

Because the unidirectional display will one day go away, and having a
user option will be an obstacle to getting rid of it.  We have been
there with unibyte buffers.

bug#9571: 24.0.50; user option to turn off bidi, please

[Lars Magne Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Because the unidirectional display will one day go away, and having a
> user option will be an obstacle to getting rid of it.

I agree with that sentiment, but...

> We have been there with unibyte buffers.

... can we ever get rid of unibyte buffers?  I don't really see how.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

bug#9571: 24.0.50; user option to turn off bidi, please

[Eli Zaretskii]

> From: Štěpán Němec <stepnem@gmail.com>
> Cc: 9571@debbugs.gnu.org
> Date: Fri, 23 Sep 2011 15:01:32 +0200
> 
> > I happen to be a documentation freak, and tried to do my best in
> > documenting everything related to bidi.  If you can suggest specific
> > changes to the user manual or NEWS about this, please do.  Saying in
> > the manual that some feature "means trouble" is not something we use
> > to do, because users will say "if it's trouble, why don't you fix
> > it?".  We need to find a way of saying something useful that will help
> > users nonetheless.  Suggestions are welcome.
> 
> When you take one example Lars reported recently -- a buffer where the
> current bidi code wasn't able to diagnose the paragraph direction
> properly (or something) in a relatively untypical buffer (a lot of long
> lines without paragraph breaks). It made the buffer unusable. Now, how
> would a user diagnose or solve this issue? How would the current
> documentation help him in doing so?

Documentation cannot help with this, because it was a bug.  It
happened in a situation that I never envisioned, and that took several
months to bump into, since bidi-display-reordering was turned on.  The
bug was fixed since then.

I expect users who bump into such issues, and cannot find anything to
help them in the manual, to ask for help on help-gnu-emacs or on
emacs-devel.  Then they will be provided with workarounds if they are
available, or with patches if not.  IOW, treat that as we always do
with bugs.

I also expect such issues to be extremely rare and marginal by the end
of the pretest, provided that people report the problems.

> I think unless you're pretty sure such problems are not likely to occur
> again, you should at least say (in NEWS if not in the manual; I would
> probably check the NEWS first in such situation anyway) that such things
> can happen and how to work around it

I wouldn't know what to say in a way that will be useful.  Bugs that
are reported are being fixed fairly quickly, so describing them would
be an exercise in futility.  Bugs that were not yet reported are
unknown and cannot be described.  The assumption that every single
slowdown and every display-related bug in Emacs 24 are due to the
bidirectional display engine is profoundly false, at least lately, so
saying "as soon as you see any problem whatsoever, turn off bidi"
would be crying wolf for no good reason.

IOW, I feel that the issue is blown out of proportions for reason I
cannot understand.  By and large, THERE IS NO PROBLEM.  Fear of
problems, maybe.  But no real problems, at least not reported ones.

bug#9571: 24.0.50; user option to turn off bidi, please

[Eli Zaretskii]

> From: Lars Magne Ingebrigtsen <larsi@gnus.org>
> Cc: rms@gnu.org,  9571@debbugs.gnu.org
> Date: Fri, 23 Sep 2011 16:55:22 +0200
> 
> > We have been there with unibyte buffers.
> 
> ... can we ever get rid of unibyte buffers?  I don't really see how.

Sorry for being unclear. I meant the environment variable that used to
force Emacs into unibyte operation mode, and other similar user-level
devices.

bug#9571: 24.0.50; user option to turn off bidi, please

[Drew Adams]

sn> OOH, there was a time when suggesting that the user adds 
sn> something to .emacs, like (setq-default bidi-display-reordering nil)
sn> was not considered obscure. Not everything must go through the
sn> customization interface.

First, we have _not_ suggested that to users. To suggest that would be a good
first step.

Currently, users need to figure out for themselves that the variable exists,
what it does, and that it is buffer-local. And they need to learn about
buffer-local variables and `setq-default'.

No, not every user is up on this stuff. It will not be obvious to all users how
they might turn off bidiness. _That is the point of this bug report_: give users
an option and document it clearly as turning off bidi. Make it obvious to users.

It seems clear that you, Eli, are resisting making it obvious how to turn off
this feature you implemented. Is ego mixed in there a bit, perhaps?

Let me be clear (again). Bidi support is a _good_ thing. I was the first to
respond to your announcement of it, saying "Great!". And I admire your work on
it as a real accomplishment.

But that does not mean that we should not make it very clear to users how they
can easily turn it off.

For some reason, this seems to be an emotionally charged issue for some people.
When I first suggested, in emacs-devel, giving users an easy way to turn it off,
I was practically accused (not by you, Eli) of western imperialism, l2r-ism and
nasty asciiness. Nothing could be further from the truth. I'm 100% in favor of
Emacs supporting bidirectional text.

We already have a way for users to turn bidi off, and that's good. I just want
to see it as (a) a user option and (b) clearly advertised (documented).

sn> It's certainly good enough to me, but you won't find that 
sn> advice in the current documentation either, AFAICT.

Correct. Adding it to the doc would be a good first step.

But the proper test is not whether using `setq-default' for this is good enough
for you or this or that Emacs-Lisp aficionado.  Users should be able to turn
bidi off using Customize, IMO. This is a _user_ setting. It is a _user's_ choice
whether to turn bidi support off.

ez> ??? The variable and its effect are clearly documented in both the
ez> user manual and the ELisp manual. 

Documented, yes, but not clearly. You do not state clearly that to turn off bidi
support in a given buffer you need to set the variable to nil, and to turn it
off generally you need to `setq-default' it to nil.

As Stepan (sorry about the ASCII/misspelling) replied to you:

sn> (it says the variable "controls whether text in the buffer is reordered for
display";
sn> is a user with no clue about bidi likely to understand that as "when set
sn> to nil, the text in the buffer should behave just as before the bidi
sn> feature was introduced"? I don't know, but I suspect I might not, if I
sn> hadn't seen previous comments and references to the variable and its
sn> effects on emacs-devel and other places).

ez> Which is more than I can say for
ez> the other new features introduced in Emacs 24, because documentation

I agree with Eli about that last sentence, and I applaud his long and continuing
interest in and efforts for the doc - second perhaps only to Richard's.

The bidi stuff is much more, and presumably better, documented than other Emacs
24 changes, so far. A good case in point is the (apparently still volatile)
buffer-display stuff.

ez> What else besides documentation of the variable is needed to make it
ez> clear to users how to turn a feature off?

1. Clearer doc about turning it off, as Stepan pointed out (above).

2. A user option, findable by looking for _options_, e.g. `apropos-variable
bidi' (without C-u).

If optionhood is good enough for `bidi-paragraph-direction' then it is good
enough for `bidi-display-reordering' too. A user trying `apropos-variable bidi',
looking for how to turn bidi off, should find the answer easily.

sn> Also, this is not (only) about making `bidi-display-reordering' a custom
variable.

My bug report _is_ only about providing a user option for this.

rms> It has to be easy to do, so why NOT do it?
ez>
ez> Because the unidirectional display will one day go away, and having a
ez> user option will be an obstacle to getting rid of it.

When it does go away, so will the option to disable it. The fact that it might
one day go away is no reason not to have an option NOW to disable it.

bug#9571: 24.0.50; user option to turn off bidi, please

[Stefan Monnier]

>> > We have been there with unibyte buffers.
>> ... can we ever get rid of unibyte buffers?  I don't really see how.
> Sorry for being unclear. I meant the environment variable that used to
> force Emacs into unibyte operation mode, and other similar user-level
> devices.

Aka "unibyte sessions".


        Stefan

bug#9571: 24.0.50; user option to turn off bidi, please

[Stefan Monnier]

> IOW, I feel that the issue is blown out of proportions for reason I
> cannot understand.  By and large, THERE IS NO PROBLEM.  Fear of
> problems, maybe.  But no real problems, at least not reported ones.

Agreed.

There have been a fair number of behavior bugs with respect to cursor
positioning (that's the tricky part of your changes, and IIUC the only
part that can't just be turned off by bidi-display-reordering), but they
seem to be mostly under control now (new cursor bugs are just as
likely now to exist in Emacs-23 as well).

The other source of problem has been performance, and AFAICT it's always
been linked to bidi-paragraph-direction (unsurprisingly: the rest of
the changes have almost no impact on algorithmic complexity, and even
the potentially large slowdown on the core "step to next position"
operation is drowned in all the rest of the work we do per-position
anyway).  We can make that default to `left-to-right' if needed, but
hopefully that won't even be necessary.


        Stefan

bug#9571: 24.0.50; user option to turn off bidi, please

[Eli Zaretskii]

> From: "Drew Adams" <drew.adams@oracle.com>
> Cc: <9571@debbugs.gnu.org>, <lekktu@gmail.com>
> Date: Fri, 23 Sep 2011 09:09:56 -0700
> 
> sn> OOH, there was a time when suggesting that the user adds 
> sn> something to .emacs, like (setq-default bidi-display-reordering nil)
> sn> was not considered obscure. Not everything must go through the
> sn> customization interface.
> 
> First, we have _not_ suggested that to users.

Because it's _not_ a user variable.  The fact that it's documented in
the user manual is already more than we normally would do about such
toggles.

> Currently, users need to figure out for themselves that the variable exists,
> what it does, and that it is buffer-local. And they need to learn about
> buffer-local variables and `setq-default'.

No, they aren't required to.  They should report any abnormal behavior
as bugs.

> It seems clear that you, Eli, are resisting making it obvious how to turn off
> this feature you implemented. Is ego mixed in there a bit, perhaps?

I cannot be a shrink to myself; maybe it is.  If so, it would only be
human.  I could ask you the same question.  Neither will get us
anywhere nearer to an agreement.  Just drop it.

> But that does not mean that we should not make it very clear to users how they
> can easily turn it off.

Let me be very clear: there is no way for users to turn this off.
There's an internal variable that bypasses most of the bidi-aware
code.  That variable exists for 2 purposes only: (a) let interested
individuals, such as myself, see if some problem could be due to the
bidi-aware parts of the display engine, and (b) allow Lisp code to
turn off bidi reordering where it is needed.  I don't mind you or
others take advantage of this variable if they so prefer.  But please
understand that making a user option to force Emacs use Emacs 23
vintage display code will need much more work than just adding a
defcustom, because that code was extensively modified as part of
development of the bidi-aware display.

IOW, when you set bidi-display-reordering to nil, all bets are off
regarding the correctness of the display operation, because I do that
only very rarely (lately almost never), and only for specific
debugging purposes.  I cannot in good faith recommend that users use
this mode because I don't use it enough to vouch for its correctness
and its being free of bugs.  You are on your own when you use this
mode.  Bugs reported for Emacs 24 when this variable is nil will go to
the bottom of my todo list.

So by pressing for this user option you actually do them a disservice.
I realize that you do that because you are not aware of the extent of
changes done to the Emacs 23 display code.  That's why I say this: now
you do know.

> We already have a way for users to turn bidi off, and that's good.

No, we don't.  What you call "bidi" cannot be turned off, not
entirely.  For example, there's no way to turn off reordering of the
mode line and header line, without switching off the reordering
entirely.  Likewise for the tool bar.  It is unthinkable for
user-friendly settings to behave like that.

> But the proper test is not whether using `setq-default' for this is good enough
> for you or this or that Emacs-Lisp aficionado.  Users should be able to turn
> bidi off using Customize, IMO. This is a _user_ setting. It is a _user's_ choice
> whether to turn bidi support off.

But it is _not_ a user setting.  It was never meant to be, and the
code was neither designed nor implemented with such an option in mind.
It's not just a missing defcustom, much more is missing to make Emacs
display dual-mode like what you seem to think.  IOW, your mental model
of this variable and its effects is profoundly wrong.

> ez> ??? The variable and its effect are clearly documented in both the
> ez> user manual and the ELisp manual. 
> 
> Documented, yes, but not clearly. You do not state clearly that to turn off bidi
> support in a given buffer you need to set the variable to nil, and to turn it
> off generally you need to `setq-default' it to nil.

Because it's not a user option.  Variables that are not user options
are normally not documented at all in the user manual.  I went out of
my way in this matter.  More information about this (including the
effect of setq-default for this variable) can be found in the ELisp
manual, and I wrote it there because I consider that manual to be
important for Emacs maintainers, not just for ELisp programmers.

> If optionhood is good enough for `bidi-paragraph-direction' then it is good
> enough for `bidi-display-reordering' too.

The paragraph direction is (and must be) a user setting.  Only a human
can know better than Emacs what is the correct base direction of the
paragraphs in a document.  Other bidi-aware programs all have
equivalent knobs.  But no other application I know of has an option to
turn off bidi reordering.  They either do it always or not at all.
That's how Emacs's bidi display was designed, too.  You are asking for
something that is not in the design.

> rms> It has to be easy to do, so why NOT do it?
> ez>
> ez> Because the unidirectional display will one day go away, and having a
> ez> user option will be an obstacle to getting rid of it.
> 
> When it does go away, so will the option to disable it. The fact that it might
> one day go away is no reason not to have an option NOW to disable it.

That is a naive and unrealistic view of software maintenance.  Users
who will customize this option on their .emacs file will object to the
change, and rightfully so.  The result will be slow and painful
process that will take years.  We have been there more than once.  For
once, I'd like to try doing this The Right Way, where things depend on
me.  Maybe it's a lost battle, but I'd like to try fighting it anyway.

bug#9571: 24.0.50; user option to turn off bidi, please

[Lars Magne Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Sorry for being unclear. I meant the environment variable that used to
> force Emacs into unibyte operation mode, and other similar user-level
> devices.

Yeah, that wasn't very helpful in the long term.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

bug#9571: 24.0.50; user option to turn off bidi, please

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Štěpán Němec <stepnem@gmail.com>,
>   9571@debbugs.gnu.org,  lekktu@gmail.com
> Date: Fri, 23 Sep 2011 13:46:35 -0400
> 
> cursor positioning [is] the tricky part of your changes, and IIUC
> the only part that can't just be turned off by bidi-display-reordering

By and large, yes.  But cursor positioning is very central to user
experience.  And there are other, less major pieces of the display
engine that were modified without keeping the old code conditioned on
bidi-display-reordering.  I never considered it a goal to keep the old
display code intact, so I cannot guarantee I did, and I know for a
fact that some places other than cursor positioning have unconditional
changes.

> The other source of problem has been performance, and AFAICT it's always
> been linked to bidi-paragraph-direction

That is one potentially expensive part of the design.  There's
another: searching for portions of text covered by "replacing" display
properties (because those text parts are reordered for display as a
single unit).  Both issues are kept at bay by limiting the amount of
text we search before giving up.

That said, the above two performance are explicitly present in the
design.  There are others that are unintended (a.k.a. "bugs").
Lately, more often than not, I find that slowdown is due to those
unintended factors, not to the above 2 inherently expensive design
traits.  Bug reports with details of the use case are necessary to
find these and weed them out.

bug#9571: 24.0.50; user option to turn off bidi, please

[Drew Adams]

> > sn> OOH, there was a time when suggesting that the user adds 
> > sn> something to .emacs, like (setq-default 
> > sn> bidi-display-reordering nil)
> > sn> was not considered obscure. Not everything must go
> > sn> through the customization interface.
> > 
> > First, we have _not_ suggested that to users.
> 
> Because it's _not_ a user variable.

Not yet.  That's precisely the bug that this thread is about.

> > Currently, users need to figure out for themselves that the 
> > variable exists, what it does, and that it is buffer-local.
> > And they need to learn about buffer-local variables and
> > `setq-default'.
> 
> No, they aren't required to.

If they want to turn off bidi support, they are.
That's what this thread is about.

> > But that does not mean that we should not make it very 
> > clear to users how they can easily turn it off.
> 
> Let me be very clear: there is no way for users to turn this off.
> There's an internal variable that bypasses most of the bidi-aware
> code.

How is that different from turning it off, from a user perspective?

Please keep a user, not just an implementor, point of view here.

> That variable exists for 2 purposes only: (a) let interested
> individuals, such as myself, see if some problem could be due to the
> bidi-aware parts of the display engine, and (b) allow Lisp code to
> turn off bidi reordering where it is needed.  I don't mind you or
> others take advantage of this variable if they so prefer.

Thank you.

> But please understand that making a user option to force
> Emacs use Emacs 23 vintage display code will need much more
> work than just adding a defcustom, because that code was
> extensively modified as part of development of the
> bidi-aware display.

So maybe it needs more time.

> IOW, when you set bidi-display-reordering to nil, all bets
> are off regarding the correctness of the display operation,

So maybe it needs more time to become fully baked.

> because I do that only very rarely (lately almost never),
> and only for specific debugging purposes.  I cannot in
> good faith recommend that users use this mode because I
> don't use it enough to vouch for its correctness
> and its being free of bugs.

More time needed to test and make it correct, it sounds like.

> You are on your own when you use this mode.

Which means you are on your own when you use Emacs 24.

Oh, sure, you'll support bidi on, but not bidi off.
And anyone who wants it off can just use Emacs 23...

> Bugs reported for Emacs 24 when this variable is nil
> will go to the bottom of my todo list.

Clear enough, I guess.  You closed this bug with lightning speed.  If Richard
hadn't chimed in, I suppose it would have died instantly.

> So by pressing for this user option you actually do
> them a disservice.

No, I think I do "them" a service.  By refusing to make it a user option and
refusing to debug and support the nil case, I think you do "them" a disservice.

Just one opinion.  And yes, a completely _naive_ opinion wrt the
design/implementation.

You've essentially said that there can be no other design/implementation that
lets users turn this off cleanly.  If that's true then OK, there's nothing more
I can say about this.

> I realize that you do that because you are not aware of
> the extent of changes done to the Emacs 23 display code.

The latter is certainly true.  But that is not the reason why I "do that".

You are taking an implementor point of view, and you are the expert there.  I
have nothing to say about that.  I am taking a user point of view  (and speaking
for only one user), ignorant of the design.

If design/implementation concerns mean that users can have no option about bidi,
then so be it.  I can't speak to that.  That's your call.  So far, however, I
see `bidi-display-reordering', and my understanding is that it gives users some
control.

> That's why I say this: now you do know.

Yes, it's clear that you will support the new feature, but not turning it off.
To me, that's unfortunate.  But maybe it's the best we can hope for.

> > We already have a way for users to turn bidi off, and that's good.
> 
> No, we don't.  What you call "bidi" cannot be turned off, not
> entirely.  For example, there's no way to turn off reordering of the
> mode line and header line, without switching off the reordering
> entirely.  Likewise for the tool bar.  It is unthinkable for
> user-friendly settings to behave like that.

OK, so a user cannot turn it off completely.  It would be good to document that,
e.g., in the doc for `bidi-display-reordering': just what it does and does not
affect.

But to the extent that they _can_ turn it off, there is apparently a way:
`bidi-display-reordering'.

> > But the proper test is not whether using `setq-default'
> > for this is good enough for you [SN] or this or that
> > Emacs-Lisp aficionado.  Users should be able to turn
> > bidi off using Customize, IMO. This is a _user_ setting. It 
> > is a _user's_ choice whether to turn bidi support off.
> 
> But it is _not_ a user setting.

Precisely what this bug thread is about.  Please make it a _user_ setting, if
you can.  That's the request.

> It was never meant to be, and the code was neither designed
> nor implemented with such an option in mind.
> It's not just a missing defcustom, much more is missing to
> make Emacs display dual-mode like what you seem to think.

So it needs more work, it sounds like.

> IOW, your mental model of this variable and its effects
> is profoundly wrong.

You've clarified things for my mental model.  Now can we please fix the design
so it DTRT?  Let users turn it off properly, if you can.

> > ez> ??? The variable and its effect are clearly documented 
> > ez> in both the user manual and the ELisp manual. 
> > 
> > Documented, yes, but not clearly. You do not state
> > clearly that to turn off bidi support in a given buffer
> > you need to set the variable to nil, and to turn it
> > off generally you need to `setq-default' it to nil.
> 
> Because it's not a user option.  Variables that are not user options
> are normally not documented at all in the user manual.  I went out of
> my way in this matter.  More information about this (including the
> effect of setq-default for this variable) can be found in the ELisp
> manual, and I wrote it there because I consider that manual to be
> important for Emacs maintainers, not just for ELisp programmers.

See above.  It _should_ be a user option, if possible.  That's the point of this
thread - see the Subject line.

Even in its current state, the doc is not helpful enough to users.  See above.
Saying that the doc should not help users because this is not a user option is,
well,...

> You are asking for something that is not in the design.

I believe you.

> > rms> It has to be easy to do, so why NOT do it?
> > ez>
> > ez> Because the unidirectional display will one day go 
> > ez> away, and having a user option will be an obstacle
> > ez> to getting rid of it.
> > 
> > When it does go away, so will the option to disable it.
> > The fact that it might one day go away is no reason not
> > to have an option NOW to disable it.
> 
> That is a naive and unrealistic view of software maintenance.

Why would a user option that no longer has any effect not be removed?  We've
done that lots of times.

> Users who will customize this option

Glad to hear you thinking in terms of that possibility.

> on their .emacs file will object to the change, and
> rightfully so.

Maybe, maybe not.  Depends what the non-nil state of things is at that point,
and depends on user needs and expectations at that point.  With your expected
scenario, everyone and her brother will _want_ to use non-nil anyway, so no one
would object.

Anyway, Emacs Dev has lots of times made changes over user objections.
Especially if there are very few objectors (which is what you and I both
expect), that's not a problem.

> The result will be slow and painful process that will
> take years.  We have been there more than once.

Sorry, I don't see it that way.  That sounds a bit like scare tactics, to me.
You might be right, or not.  I don't see the sky falling because Emacs Dev wants
to remove a user option that no longer has any effect.

> For once, I'd like to try doing this The Right Way,

Great.  Then let's please take the time to finish things in such a way that
users can completely and easily turn this on/off.  If you can.

> where things depend on me.  Maybe it's a lost battle,
> but I'd like to try fighting it anyway.

Sounds more like a won battle, to me.  You seem to say that only the current
design is possible; it's impossible to cleanly let users turn it off; if users
do try to turn it off that won't be supported;...  In sum, your message to users
seems to be, "If you don't need or want bidi then use Emacs 23."

bug#9571: 24.0.50; user option to turn off bidi, please

[Eli Zaretskii]

> Date: Fri, 23 Sep 2011 21:24:03 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 9571@debbugs.gnu.org, lekktu@gmail.com, stepnem@gmail.com
> 
> > cursor positioning [is] the tricky part of your changes, and IIUC
> > the only part that can't just be turned off by bidi-display-reordering
> 
> By and large, yes.  But cursor positioning is very central to user
> experience.  And there are other, less major pieces of the display
> engine that were modified without keeping the old code conditioned on
> bidi-display-reordering.

I remembered another important piece of code that doesn't pay
attention to bidi-display-reordering: mouse highlight.  It was
completely reimplemented with bidi-aware code.

bug#9571: 24.0.50; user option to turn off bidi, please

[Richard Stallman]

    > If the reordering has confusing effects, turning off bidi display
    > would enable you to see clearly what the real order is in the buffer.

    As I said (following Stefan), find-file-literally has that effect.

It has a lot of other effects too, which are painful except when you
happen to want them.  That is no substitute for what I'm suggesting.

Why are you opposed to a flag to turn bidi display off?

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use free telephony http://directory.fsf.org/category/tel/

bug#9571: 24.0.50; user option to turn off bidi, please

[Richard Stallman]

    What confusion are you referring to?

Not being sure about the order, in the buffer, of the characters you see
on the screen.

					  Are we talking about a buffer
    with or without R2L characters?

It would have to be a case where characters that have bidi
significance are present.  If there are none, there is no reordering,
thus no possibility it can cause confusion.

				     And how would setting
    bidi-display-reordering to nil resolve that confusion?

You would see all the characters in the order that they
appear in the buffer.

I don't envision anyone would want to edit this way.  But it could be
useful to look at this kind of display once in a while.

    Because the unidirectional display will one day go away, and having a
    user option will be an obstacle to getting rid of it.

Why should it ever be deleted?  What is gained by deleting it?


-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use free telephony http://directory.fsf.org/category/tel/

bug#9571: 24.0.50; user option to turn off bidi, please

[Eli Zaretskii]

> From: "Drew Adams" <drew.adams@oracle.com>
> Cc: <stepnem@gmail.com>, <9571@debbugs.gnu.org>, <lekktu@gmail.com>
> Date: Fri, 23 Sep 2011 12:03:57 -0700
> 
> > Let me be very clear: there is no way for users to turn this off.
> > There's an internal variable that bypasses most of the bidi-aware
> > code.
> 
> How is that different from turning it off, from a user perspective?

_Parts_of_it_ can be turned off.  What you get is part bidi and part
not.  What that does is anyone's guess.

> > But please understand that making a user option to force
> > Emacs use Emacs 23 vintage display code will need much more
> > work than just adding a defcustom, because that code was
> > extensively modified as part of development of the
> > bidi-aware display.
> 
> So maybe it needs more time.

That time will help only if someone will work on developing that
mode.  I won't any time soon; volunteers are welcome.

> > Bugs reported for Emacs 24 when this variable is nil
> > will go to the bottom of my todo list.
> 
> Clear enough, I guess.  You closed this bug with lightning speed.

Please get your facts right.  The bug is not closed.  I never closed
it.

> > So by pressing for this user option you actually do
> > them a disservice.
> 
> No, I think I do "them" a service.  By refusing to make it a user option and
> refusing to debug and support the nil case, I think you do "them" a disservice.

Anything can be turned on its head by clever word juggling.  But the
truth is still there: given the current design and implementation of
the bidi display, you are encouraging them to use unsupported mode,
whereas I encourage them to use a mode that is fully supported and
whose bugs are being fixed in a matter of days if not hours.

> You've essentially said that there can be no other design/implementation that
> lets users turn this off cleanly.

There _could_ be a different design, but it's too late to talk about
that now, that the existing design and implementation are what they
are and my free time, age, and amount of energy are what they are.

> You are taking an implementor point of view, and you are the expert there.  I
> have nothing to say about that.  I am taking a user point of view  (and speaking
> for only one user), ignorant of the design.

Ignorance is not always a bliss.  I took time to explain the
implementation because (I hope) those explanations can help you and
others understand why pressing for making this a user variable is not
TRT, in practical terms.  Armed with this new knowledge, I trust
interested users, including you, to draw their conclusions.

> If design/implementation concerns mean that users can have no option about bidi,
> then so be it.  I can't speak to that.  That's your call.  So far, however, I
> see `bidi-display-reordering', and my understanding is that it gives users some
> control.

It's only a partial control, and the result is a weird creature that
is neither pre-Emacs 24 display nor full bidi-aware display.  IOW, the
result is incoherent.  It will probably work most of the time, but I
cannot bet on that.  Whether you want to use such a creature is your
call now, that I explained the meaning as clear as I could.

> OK, so a user cannot turn it off completely.  It would be good to document that,
> e.g., in the doc for `bidi-display-reordering': just what it does and does not
> affect.

So now we are requested to document deeply internal details of the
current implementation, and in the user manual at that?  Do you
understand what kind and amount of highly technical stuff will have to
be in the manual in order to explain this in enough detail for users
to understand it?  How far can you go ad absurdum, Drew?

> But to the extent that they _can_ turn it off, there is apparently a way:
> `bidi-display-reordering'.

That way lies madness, if you ask me.  At least in the long run, if
not today.  You are free to go there, of course: it's a free world
(most of it).

> > But it is _not_ a user setting.
> 
> Precisely what this bug thread is about.  Please make it a _user_ setting, if
> you can.  That's the request.

It doesn't take a bidi expert to add a defcustom.  I will object to
that, for the reasons I explained, and I certainly won't do it
myself.  But I cannot force others not to do that.

> > It was never meant to be, and the code was neither designed
> > nor implemented with such an option in mind.
> > It's not just a missing defcustom, much more is missing to
> > make Emacs display dual-mode like what you seem to think.
> 
> So it needs more work, it sounds like.

If we can find someone to invest that work.

> > IOW, your mental model of this variable and its effects
> > is profoundly wrong.
> 
> You've clarified things for my mental model.  Now can we please fix the design
> so it DTRT?  Let users turn it off properly, if you can.

I have more important things on my table wrt bidi support, and not
enough time.  So I won't be working on this in the near future, sorry.

> In sum, your message to users seems to be, "If you don't need or
> want bidi then use Emacs 23."

Please don't put your words in my mouth.  Emacs 24 is quite usable as
it is, and there's no need for users to stay with Emacs 23.  At least
not users who approach this issue rationally.

bug#9571: 24.0.50; user option to turn off bidi, please

[Eli Zaretskii]

> Date: Fri, 23 Sep 2011 15:44:31 -0400
> From: Richard Stallman <rms@gnu.org>
> CC: drew.adams@oracle.com, 9571@debbugs.gnu.org
> 
> Why are you opposed to a flag to turn bidi display off?

I explained that at length in followups.

bug#9571: 24.0.50; user option to turn off bidi, please

[Stefan Monnier]

>> Because it's _not_ a user variable.
> Not yet.  That's precisely the bug that this thread is about.

I suggest to drop this useless thread.  It's not a user option and won't
be.  Ideally we'll remove it altogether before Emacs-24.1 is released
(or we may just rename it to `bidi--fiddle-with-internals').  Making it
more user-visible is not on the radar (BTW, Eli, I think it should be
removed from the user manual).


        Stefan

bug#9571: 24.0.50; user option to turn off bidi, please

[Eli Zaretskii]

> Date: Fri, 23 Sep 2011 15:44:34 -0400
> From: Richard Stallman <rms@gnu.org>
> CC: monnier@iro.umontreal.ca, 9571@debbugs.gnu.org
> 
>     What confusion are you referring to?
> 
> Not being sure about the order, in the buffer, of the characters you see
> on the screen.

If you need to see the buffer order, moving with C-f and C-b will show
you the order in the buffer, because cursor motion still works in
buffer order.

> 					  Are we talking about a buffer
>     with or without R2L characters?
> 
> It would have to be a case where characters that have bidi
> significance are present.

If the reordering is correct, there could be no confusion, because the
text is then legible.  If it is illegible, then there's a bug that
needs to be reported.

For example, if there are no R2L characters in the text, the result of
reordering should be identical to not reordering at all.  Any person
who can read whatever language we are talking about will instantly
distinguish between correct and incorrect order.  I see no place for
any confusion here.

Users will gain nothing from this option, because turning off
reordering when R2L text is involved does not make the text more
legible; quite the opposite.

I explained in my other messages what will users lose if they have
such an option, unless someone invests a non-trivial amount of work to
restore the old code where I deleted or rewrote it, and maintain that
code as a separate execution branch for years to come.

>                 If there are [no R2L characters], there is no
> reordering, thus no possibility it can cause confusion.

You are wrong: _all_ characters are displayed in Emacs 24 via the
reordering engine.  It's just that plain left-to-right text emerges
from that reordering in its original buffer order.  But the reordering
engine doesn't "know" that, it just implements the rules of
reordering.

> 				     And how would setting
>     bidi-display-reordering to nil resolve that confusion?
> 
> You would see all the characters in the order that they
> appear in the buffer.

That will not resolve any confusion.  As someone who happened to read
R2L text in Emacs 23 (e.g., in email messages), I can assure you:
seeing R2L text in buffer order confuses even more than seeing results
of slightly incorrect reordering.  It makes the reading process very
slow and error prone, even if your command of the language is very
good.

> I don't envision anyone would want to edit this way.  But it could be
> useful to look at this kind of display once in a while.

It's not useful for users, believe me.  It could be useful to someone
who debugs Emacs display, but there's no need for user option for
that use case.

>     Because the unidirectional display will one day go away, and having a
>     user option will be an obstacle to getting rid of it.
> 
> Why should it ever be deleted?  What is gained by deleting it?

Easier maintenance.  Emacs display engine is already more complex that
humanly perceptible.  Having two divergent engines in one means
unnecessarily complicating maintenance and slowing down development,
because every change needs to be tested twice and every new feature
needs to be designed so it works in both modes.  I'm quite sure doing
this will be a waste of resources which are already at premium.
There's no need for the old unidirectional display code; as I
explained above, straight left-to-right text is treated by the
bidirectional code correctly and transparently to the users.

bug#9571: 24.0.50; user option to turn off bidi, please

[Drew Adams]

> > > Let me be very clear: there is no way for users to turn this off.
> > > There's an internal variable that bypasses most of the bidi-aware
> > > code.
> > 
> > How is that different from turning it off, from a user perspective?
> 
> _Parts_of_it_ can be turned off.  What you get is part bidi and part
> not.  What that does is anyone's guess.

Anyone's guess, unless someone checks and specifies what it does.
And why is it that that won't happen?

> > > But please understand that making a user option to force
> > > Emacs use Emacs 23 vintage display code will need much more
> > > work than just adding a defcustom, because that code was
> > > extensively modified as part of development of the
> > > bidi-aware display.
> > 
> > So maybe it needs more time.
> 
> That time will help only if someone will work on developing that
> mode.  I won't any time soon; volunteers are welcome.

I agree.  Bidi support - being able to turn it on, is welcome.
Being able to turn it OFF is also welcome.

> > > Bugs reported for Emacs 24 when this variable is nil
> > > will go to the bottom of my todo list.
> > 
> > Clear enough, I guess.  You closed this bug with lightning speed.
> 
> Please get your facts right.  The bug is not closed.  I never closed
> it.

Sorry; I'm not very clear on the status codes/terminology.
This is the fact I was referring to:

   tags 9571 + wontfix

So I guess you're saying that it is not closed but it won't be fixed.  Nuance.
Limbo is apparently no more (and perhaps never was), but Wontfix is alive and
well.

> > > So by pressing for this user option you actually do
> > > them a disservice.
> > 
> > No, I think I do "them" a service.  By refusing to make it 
> > a user option and refusing to debug and support the nil case,
> > I think you do "them" a disservice.
> 
> Anything can be turned on its head by clever word juggling.

What word juggling?  Are you now saying that you do _not_ refuse to debug the
nil case and you _will_ support it?  I was trying to state your position
accurately, but I'll be happy to hear if I misrepresented it.

> But the truth is still there: given the current design and
> implementation of the bidi display,

I never said you could not modify the implementation.  On the contrary.  Based
on your statement that the implementation does not support the nil case yet, I
encourage work on fixing that.

> you are encouraging them to use unsupported mode,

Not at all.  Emacs 24 has not yet been released.
We still have a chance to get it right.

Until Emacs 24 has been released, talking about supported/unsupported mode
simply expresses your desire that you not support the nil case.

> whereas I encourage them to use a mode that is fully supported and
> whose bugs are being fixed in a matter of days if not hours.

I too am happy to see them use the bidi-ON mode.
And I am happy to see your level of support for it.

> > You've essentially said that there can be no other 
> > design/implementation that lets users turn this off cleanly.
> 
> There _could_ be a different design, but it's too late to talk about
> that now, that the existing design and implementation are what they
> are and my free time, age, and amount of energy are what they are.

The release is not out yet.  If the cake is not fully baked, maybe it shouldn't
be served quite yet.

> > You are taking an implementor point of view, and you are 
> > the expert there.  I have nothing to say about that.  I am taking
> > a user point of view  (and speaking for only one user), ignorant
> > of the design.
> 
> Ignorance is not always a bliss.  I took time to explain the
> implementation because (I hope) those explanations can help you and
> others understand why pressing for making this a user variable is not
> TRT, in practical terms.  Armed with this new knowledge, I trust
> interested users, including you, to draw their conclusions.

So far, it seems (but I can't speak for him, obviously) that Richard is not
convinced either.  He has repeated the same thing as I: why shouldn't this be a
user option?

And he adds, "Why should [unidirectional display] ever be deleted? What is
gained by deleting it?"  I have no opinion about that.  My only concern is that
users be able to, and know how to, turn bidirectional display off.

I mention Richard here because he is more likely than I to listen to and
understand correctly the explanations about the design.

> > If design/implementation concerns mean that users can have 
> > no option about bidi, then so be it.  I can't speak to that.
> > That's your call.  So far, however, I see `bidi-display-reordering',
> > and my understanding is that it gives users some control.
> 
> It's only a partial control, and the result is a weird creature that
> is neither pre-Emacs 24 display nor full bidi-aware display.

Emacs has been about partial control, better-than-nothing, and
do-the-best-we-can, since its inception.  Above all, it has been about giving
users as much control as possible.

And about making things transparent (clear) to users, including about how
partial the control might be.  That, in particular, is not something we have
hidden from users - unlike the case for some non-free software.

> IOW, the result is incoherent. It will probably work most of
> the time,

And that's about as good as one can say for most of Emacs.

> but I cannot bet on that.

No one is asking you to bet.  This bug report only asks that you make the
variable, which already exists, which already gives users "partial control", and
which already "probably work[s] most of the time", into a _user option_.  And
that you clearly document how to turn things off (so far as that can be done).

No one is asking more than that.  If you do not want to, or we do not have the
resources to, make the nil case work perfectly, then we will anyway live with it
being imperfect.

It's about giving users the knowledge and access, even if the results of using
nil are not 100% predictable or 100% good.

> Whether you want to use such a creature is your
> call now, that I explained the meaning as clear as I could.

It's not about me.  It has never been about me.  I might turn it on or I might
turn it off.  If I get the impression that it slows things down I might try
turning it off.  If I get the impression that it has no negative effect I might
turn it on.

I now know how to turn it off (as much as is possible), thanks to the knowledge
you communicated - here and in the manual.  But other users might not know that.

I would prefer that we offer a user option.  Not for me, but for others, who
might not be so clear about Lisp, buffer-local variables, and `setq-default'.

> > OK, so a user cannot turn it off completely.  It would be 
> > good to document that, e.g., in the doc for
> > `bidi-display-reordering': just what it does and does not
> > affect.
> 
> So now we are requested to document deeply internal details of the
> current implementation, and in the user manual at that?

That's you saying that, not I.  Here is your text I was responding to, in fact:

>>> For example, there's no way to turn off reordering of the
>>> mode line and header line, without switching off the reordering
>>> entirely.  Likewise for the tool bar.  It is unthinkable for
>>> user-friendly settings to behave like that.

Unthinkable that the settings behave like that - maybe.  But what's quite
thinkable is that users of the variable that you documented might want to know
that it does not turn off reordering of the mode line etc.

Telling them that - just what you told us here, is not "document[ing] deeply
internal details of the current implementation" - please don't exaggerate.  That
is simply presenting a caveat, documenting a few special cases so users don't
expect something different in those cases.

> Do you understand what kind and amount of highly technical stuff
> will have to be in the manual in order to explain this in enough
> detail for users to understand it?

What kind and amount of "highly technical stuff" did you have to go into, to
communicate to us those few corner cases?  I didn't see _anything_ highly
technical in what you said, but what you said helped me know what to expect for
the mode line etc.  Other users deserve the same info.

> How far can you go ad absurdum, Drew?

You are the one stretching things, Eli.  You give us a sentence that makes clear
not to expect turn-off of reorder in cases a, b, and c.  When I say, great,
thank you, please tell that to the users also, you freak out and start screaming
too "highly technical" and "deeply internal" for users. Ad absurdum, indeed.

No one is asking that you document the implementation.  Think in terms of what
might help a user who does happen to set the variable to nil.  That's the kind
of info that it could be helpful to add to the manual.  Nothing more.

> > But to the extent that they _can_ turn it off, there is 
> > apparently a way: `bidi-display-reordering'.
> 
> That way lies madness, if you ask me.  At least in the long run, if
> not today.  You are free to go there, of course: it's a free world
> (most of it).

So you introduced, and documented in the user manual, something that leads to
madness?  That's not my doing.

> > > But it is _not_ a user setting.
> > 
> > Precisely what this bug thread is about.  Please make it a 
> > _user_ setting, if you can.  That's the request.
> 
> It doesn't take a bidi expert to add a defcustom.  I will object to
> that, for the reasons I explained, and I certainly won't do it
> myself.  But I cannot force others not to do that.

What do you call tagging the request for that as "wontfix"?
 
> > > It was never meant to be, and the code was neither designed
> > > nor implemented with such an option in mind.
> > > It's not just a missing defcustom, much more is missing to
> > > make Emacs display dual-mode like what you seem to think.
> > 
> > So it needs more work, it sounds like.
> 
> If we can find someone to invest that work.

We agree that would be a good thing.  So at least this bug should be classified
"wishlist" instead of "wontfix".

> > > IOW, your mental model of this variable and its effects
> > > is profoundly wrong.
> > 
> > You've clarified things for my mental model.  Now can we 
> > please fix the design so it DTRT?  Let users turn it off
> > properly, if you can.
> 
> I have more important things on my table wrt bidi support, and not
> enough time.  So I won't be working on this in the near future, sorry.

Yes, you made that clear with your first reply.

In fact, you quickly made it clear that no one else would work on fixing this
either: "tags 9571 + wontfix"

> > In sum, your message to users seems to be, "If you don't need or
> > want bidi then use Emacs 23."
> 
> Please don't put your words in my mouth.  Emacs 24 is quite usable as
> it is, and there's no need for users to stay with Emacs 23.  At least
> not users who approach this issue rationally.

Then just what words would you like to use to end the sentence "If you don't
need or want bidi then use..."?  Don't let me put words in your mouth - you tell
us how the sentence ends, please.

bug#9571: 24.0.50; user option to turn off bidi, please

[Juanma Barranquero]

On Fri, Sep 23, 2011 at 23:23, Drew Adams <drew.adams@oracle.com> wrote:

> Anyone's guess, unless someone checks and specifies what it does.
> And why is it that that won't happen?

You remind me of a discussion in the Perl 6 mailing list, a few years ago:

  "As one of the 3 or 4 people in the world that understands the perl5
regexp engine, we would welcome your input."
  "'Understands' is a rather strong word..."
                              -- Mark Kvale and Hugo van der Sanden

And that is the case here: the number of people who understands both
the Emacs display engine and the issues related to bidi can be counted
with a couple bits, and I'm being generous. So if Eli says that he's
not going to devote time to some issue related to it, it's not
unrealistic to expect that it won't happen (soonish). Your insistence
in keeping the issue as a wishlist is just the hope that someone will
someday want to change the display engine to suit your preferences.
Even if the number of knowledgeable people suddenly were to increase,
that wouldn't automatically mean that they would be willing to invest
in adding the code and the toggle to keep the display engine
backward-compatible.

> Limbo is apparently no more

Not exactly, no. In fact, the official position of the Catholic Church
has not changed, news reports notwithstanding.

http://en.wikipedia.org/wiki/Limbo#Modern_era

> Not at all.  Emacs 24 has not yet been released.
> We still have a chance to get it right.

Perhaps if you can convince Eli, Stefan and Chong that supporting both
the bidi-aware and the not-bidi-aware code is the "right" thing to do.
If you can do that, please also convince them to simultaneously
support the Unicode and pre-Unicode font backends; I miss the speed of
my line-by-line scrolling in times past (though it is now quite
usable, thanks to tireless efforts by Eli and Chong and Jason, IIRC).

> So far, it seems (but I can't speak for him, obviously) that Richard is not
> convinced either.  He has repeated the same thing as I: why shouldn't this be a
> user option?

Because the option it would currently offer is bogus.

> Emacs has been about partial control, better-than-nothing, and
> do-the-best-we-can, since its inception.  Above all, it has been about giving
> users as much control as possible.

It has also been about using resources (= developers) in the most
efficient way possible.

> It's about giving users the knowledge and access, even if the results of using
> nil are not 100% predictable or 100% good.

Are there really that many users that will want to disable
bidi-display-reordering, knowing that the result will likely be
buggier than the default? And if they do exist, do they really need a
defcustom?

> I would prefer that we offer a user option.  Not for me, but for others, who
> might not be so clear about Lisp, buffer-local variables, and `setq-default'.

A defcustom is an statement that something is a switch, and both modes
are reasonable. That is not the case here.

> When I say, great,
> thank you, please tell that to the users also, you freak out and start screaming
> too "highly technical" and "deeply internal" for users.

How did you determine the freaking out and the screaming? I'm quite interested.

> No one is asking that you document the implementation.  Think in terms of what
> might help a user who does happen to set the variable to nil.  That's the kind
> of info that it could be helpful to add to the manual.  Nothing more.

Your defcustom request can be trivially satisfied, and not a word is
needed in the manual:

(defcustom bidi-display-reordering t
  "Not a user option. Do NOT set it to nil.  Horrible things will
happen.  Thanks."
  :type 'boolean
  :version "24.1"

    Juanma

bug#9571: 24.0.50; user option to turn off bidi, please

[Drew Adams]

> And that is the case here: the number of people who understands both
> the Emacs display engine and the issues related to bidi can be counted
> with a couple bits, and I'm being generous. So if Eli says that he's
> not going to devote time to some issue related to it, it's not
> unrealistic to expect that it won't happen (soonish).

I think everyone realizes that.  This bug report is about making the variable
into a user option.  As Eli has said, we don't need his valuable time and
expertise for that.

> Your insistence in keeping the issue as a wishlist

I mentioned "wishlist" only after Eli himself indicated that it needs more work
"if we can find someone to invest that work".

I don't think this bug report - which asks for a user option, should be closed,
or classified wontfix (already done), or sent to the wishlist.

I think we should make the var an option now AND (based on what Eli has said)
add an enhancement request to the wishlist for more development to support the
nil value - whether or not Eli is the only one on the planet who could ever hope
to respond to such a wish.

If you exclude things from the wishlist just because there is no one around
today who has the time to work on them, then it is not a wishlist.

> is just the hope that someone will someday want to change
> the display engine to suit your preferences.

No, you haven't been reading what I said.  It's not about my preferences.

I truly have no idea whether I will want this thing on or off, personally, as I
indicated clearly several times.  As I said, if there seems to be no downside, I
will likely just leave it on.

That's even though I do not expect to have any need for bidirectional editing.
Why then?  To closer reflect what others use, including people who might use
some of my code.  As you know, each departure from emacs -Q makes bug reporting
just that much harder.

But again, I have no idea now what I will do, and this bug report has NOTHING to
do with my (nonexistent) preferences wrt the variable value.

> > Limbo is apparently no more
> 
> Not exactly, no. In fact, the official position of the Catholic Church
> has not changed, news reports notwithstanding.
> http://en.wikipedia.org/wiki/Limbo#Modern_era

Thinking, in fact, of you, Juanma, I read precisely that Wikipedia article
BEFORE I wrote that sentence, being personally ignorant and indifferent to
limbosity.

And that is _exactly_ why I added "(and perhaps never was)" immediately
following the words you quoted (but which you chose to cut).

That article makes it clear that the situation wrt the Catholic Church is less
than clear wrt "limbo" - even after P.B. XVI's 2007 publication, and that there
have been multiple notions of "limbo" over the ages, etc.

"And perhaps never was" sums up the situation quite accurately, as I read that
Wikipedia article.  Not that I really care...

> > So far, it seems (but I can't speak for him, obviously) 
> > that Richard is not convinced either.  He has repeated the
> > same thing as I: why shouldn't this be a user option?
> 
> Because the option it would currently offer is bogus.

Then so is the same variable as a NON-option bogus.  And then so is its
treatment in two (count 'em!) Emacs manuals.  I didn't create that variable, nor
did I describe it in the manuals.

You cannot have it both ways.  Either this is something useful for users to know
about or it is not.  If it is, then AFAICT it is just as useful for non-Lispers
as for Lispers, for those who understand `setq-default' and buffer-local vars as
well as for those who do not.

> > Emacs has been about partial control, better-than-nothing, and
> > do-the-best-we-can, since its inception.  Above all, it has 
> > been about giving users as much control as possible.
> 
> It has also been about using resources (= developers) in the most
> efficient way possible.

There are two different issues that have been discussed here:

1. Make the variable into an option, and tell users more clearly about what nil
does.

2. Work more on the code to be able to more solidly "support" the nil use case.

This bug report asked only for #1.  As Eli has admitted, it doesn't take much in
the way of resources to do that.  It is Eli, not I, who brought up #2 and
pointed to potential problems if users set the variable to nil.

And let's not be under any illusions about this "support".  Not making it a user
option does not let Emacs "support" off the hook.  The existence of the
variable, and especially its description in both Emacs manuals, means that some
users will likely set it to nil.

> > It's about giving users the knowledge and access, even if 
> > the results of using nil are not 100% predictable or 100% good.
> 
> Are there really that many users that will want to disable
> bidi-display-reordering,

Who knows?  Do you?

> knowing that the result will likely be buggier than the
> default?

But how on Earth will they know that?  Saying anything about that in the doc was
specifically verboten by Eli:

  Saying in the manual that some feature "means trouble" is
  not something we use [sic] to do, because users will say "if
  it's trouble, why don't you fix it?".

> And if they do exist, do they really need a defcustom?

To quote a famous person, why not?  "Why are you opposed to a flag to turn bidi
display off?"

Why should you, Juama, who are familiar enough with buffer-local vars etc. to
understand how to turn this off (if you wanted to), be able to do so; but not
Jane Doe, who understans how to use Customize but knows nothing about
`setq-default'?

Do you "really need" to keep this optional behavior to yourself and others with
Emacs-Lisp knowledge?

> > I would prefer that we offer a user option.  Not for me, 
> > but for others, who might not be so clear about Lisp,
> > buffer-local variables, and `setq-default'.
> 
> A defcustom is an statement that something is a switch, and both modes
> are reasonable. That is not the case here.

No more so than for any other variable.  Nothing is guaranteed in Emacs.  User
options no more than other vars.  You are making a mountain out of a mole hill.

> > No one is asking that you document the implementation.  
> > Think in terms of what might help a user who does happen
> > to set the variable to nil.  That's the kind
> > of info that it could be helpful to add to the manual.  
> > Nothing more.
> 
> Your defcustom request can be trivially satisfied, and not a word is
> needed in the manual:
> 
> (defcustom bidi-display-reordering t
>   "Not a user option. Do NOT set it to nil.  Horrible things will
> happen.  Thanks." :type 'boolean :version "24.1"

You forgot a paren.)  But I could actually live with that, provided the manual
still describes the variable fairly, as it does now (and hopefully a bit more
clearly wrt what nil does).

I wouldn't choose such a doc string myself, but if that's the price to pay for
giving users an option they can find easily using `apropos-variable' then it
might be worth it. 

But I don't think you'll convince Eli to use such language - see above.

bug#9571: 24.0.50; user option to turn off bidi, please

[Juanma Barranquero]

On Sat, Sep 24, 2011 at 02:32, Drew Adams <drew.adams@oracle.com> wrote:

> As Eli has said, we don't need his valuable time and
> expertise for that.

He's already been useful in this message thread by explaining that
setting the variable to nil does not do what people thinks it does.

> I don't think this bug report - which asks for a user option, should be closed,
> or classified wontfix (already done), or sent to the wishlist.

OK.

> I think we should make the var an option now AND (based on what Eli has said)
> add an enhancement request to the wishlist for more development to support the
> nil value - whether or not Eli is the only one on the planet who could ever hope
> to respond to such a wish.

Eli has said that he won't oppose someone adding the defcustom, so you
can do both (sending a patch to add the defcustom, and remove the
wishlist tag from the bug).

> If you exclude things from the wishlist just because there is no one around
> today who has the time to work on them, then it is not a wishlist.

Of course. I just added today a wishlist item for datagram sockets on
Windows, knowing full well that there aren't many people with the
knowledge and the inclination to spend time implementing them. But
experience of years shows that redisplay engine experts are few and
far between. We can leave this as a wishlist, it's just that it won't
likely serve any useful purpose.

> No, you haven't been reading what I said.  It's not about my preferences.

I meant, your preference of the display enginge being able to be
turned off, even if you personally wouldn't want to do it.

> Thinking, in fact, of you, Juanma, I read precisely that Wikipedia article
> BEFORE I wrote that sentence, being personally ignorant and indifferent to
> limbosity.

Why, pray tell, thinking of me? Because I'm nitpicky enough to answer
to that, or because being a Spaniard I'm more likely to be Catholic
(which I theoretically am, but not in fact)?

> And that is _exactly_ why I added "(and perhaps never was)" immediately
> following the words you quoted (but which you chose to cut).

It's not that I chose to cut these words, is that I was only replying
to the first words. So:

> That article makes it clear that the situation wrt the Catholic Church is less
> than clear wrt "limbo" - even after P.B. XVI's 2007 publication, and that there
> have been multiple notions of "limbo" over the ages, etc.

Yes. But the fact is, the position of the Catholic Church has not
changed, recently or otherwise. Belief in the limbo has traditionally
been, and still is, something that is not doctrine, but does not
contradict doctrine. That's why I commented your "Limbo is apparently
no more" and let the rest aside. I wasn't interested in discussing
limbo's theology with you, but inform you that the news reports were
far off the mark.

> "And perhaps never was" sums up the situation quite accurately, as I read that
> Wikipedia article.

Sums up the situation quite accurately. Which is unrelated to any
recent change of status.

> Then so is the same variable as a NON-option bogus.

No, if its intended use is helping in debugging.

> You cannot have it both ways.  Either this is something useful for users to know
> about or it is not.

And i don't want it both ways. I think is not useful for users to
know. Note "useful". I'm not saying that they shouldn't be able to
know it. Also note "I think". That's an opinion, I'm not claiming to
know for a fact that it is not useful for them. No one really knows.

> Who knows?  Do you?

I don't know, and neither do you. You ask for I change, I don't. So
when the masses come here to bring down the walls, we can relent and
give them the defcustom (I'm feeling generous today). Until then, it's
customizability for customizability's sake.

> But how on Earth will they know that?  Saying anything about that in the doc was
> specifically verboten by Eli:

Yes. It doesn't seem appropriate for the manual. But we have lot of
docstrings that say: "internal use only" or somesuch.

> To quote a famous person, why not?  "Why are you opposed to a flag to turn bidi
> display off?"

Because it will create the false belief that it does something useful,
so some users will mistakenly shot themselves in the foot.

If you mean: why am I opposed to making the display engine have two
code paths, one with bidi support and one without it, switchable with
a flag... I'm neither for nor against, except that it will be a lot of
work that nobody wants to do, it will make the display engine still
harder to study (I'm just repeating what Eli said a few messages ago),
and it's not clear that it will serve any useful purpose.

> Why should you, Juama, who are familiar enough with buffer-local vars etc. to
> understand how to turn this off (if you wanted to), be able to do so; but not
> Jane Doe, who understans how to use Customize but knows nothing about
> `setq-default'?

I know lots of things, for lots of reasons, that are just garbage
and/or non-useful. I don't see why should I try to help everyone know
these things. I won't try to preclude anyone to learn them, of course,
but I won't help anyone by getting them to learn to say "please" in
irish, memorize an obsolete definition of the meter, or be able to
recite the full name of the female protagonist of "The Fifth Element".
More power to Jane Doe if she wants to read the Emacs Lisp Reference
and know how to setq-default bidi-display-reordering. No one is really
going to try to stop her.

> Do you "really need" to keep this optional behavior to yourself and others with
> Emacs-Lisp knowledge?

Is that what I'm proposing? "Keeping it to myself"? I wasn't born
knowing elisp, you know; and neither had I to submit to some hermetic
mysteries' initiation to be allowed to read the manual. Quaerendo
invenietis.

> No more so than for any other variable.  Nothing is guaranteed in Emacs.  User
> options no more than other vars.  You are making a mountain out of a mole hill.

A very small mountain, perhaps. A couple cubic meters of dirt, tops.

> You forgot a paren.)

Let down by cut&paste :-(

> But I could actually live with that, provided the manual
> still describes the variable fairly, as it does now (and hopefully a bit more
> clearly wrt what nil does).

Glad to hear it.

    Juanma

bug#9571: 24.0.50; user option to turn off bidi, please

[Drew Adams]

> > Thinking, in fact, of you, Juanma, I read precisely that 
> > Wikipedia article BEFORE I wrote that sentence, being
> > personally ignorant and indifferent to limbosity.
> 
> Why, pray tell, thinking of me? Because I'm nitpicky enough to answer
> to that, or because being a Spaniard I'm more likely to be Catholic
> (which I theoretically am, but not in fact)?

Because you are knowledgable in many historical and language matters and are
wont to refer to Wikipedia.

I read that article mainly because I had heard only vague things about limbo and
wanted to get a little background before saying something completely incorrect,
even if I was really only joking about the popular (and no doubt inaccurate)
meaning of "limbo".

Thinking of you in that regard was secondary, and only because, as I say, you
are one who is both knowledgable and wont to correct inaccuracies even in
non-technical matters.  And yes, you can properly take that as a compliment,
from someone who is also sometimes interested in non-software topics.

> > "And perhaps never was" sums up the situation quite 
> > accurately, as I read that Wikipedia article.
> 
> Sums up the situation quite accurately. Which is unrelated to any
> recent change of status.

We do agree, I fear.

bug#9571: 24.0.50; user option to turn off bidi, please

[Jason Rumney]

"Drew Adams" <drew.adams@oracle.com> writes:

> 1. That's not very user-friendly.  We should have a user option that
> does this, i.e., gives users an easy way to disable this feature if they
> don't want to use it.

This report is a bit like the reports we received when 20.1 was entering
pre-release which resulted in the addition of --unibyte and related
kludges, which lived on too long and caused all sorts of bugs.

If you don't want to use bidi, just don't type or look at RTL text.

If it is slowing Emacs down on LTR text in a way that setting
bidi-display-reordering to nil fixes, then please file a bug report, I
am sure an optimisation can be made to fix such slowdowns.

bug#9571: 24.0.50; user option to turn off bidi, please

[Jason Rumney]

Richard Stallman <rms@gnu.org> writes:

>     "Figure out what's really present" is sufficiently vague that I don't
>     know what is the right way to address this need.  find-file-literally is
>     at least as likely to be effective, and in many more cases.
>
> find-file-literally disables character set decoding.  That means you'd
> have to manually figure out how the bytes correspond to the characters
> of the file, or else give a command to decode it.
>
> If the confusion has to do with bidi, the feature you want for
> understanding it is to turn off bidi and nothing else.  It has to be
> easy to do, so why NOT do it?

If you can read the relevant scripts, then bidi is not going to confuse
you. If you cannot read the scripts, then turning off bidi is not really
going to help (probably find-file-literally would be a better option in
this case).

bug#9571: 24.0.50; user option to turn off bidi, please

[Eli Zaretskii]

> From: "Drew Adams" <drew.adams@oracle.com>
> Cc: "'Eli Zaretskii'" <eliz@gnu.org>, <stepnem@gmail.com>,
>         <9571@debbugs.gnu.org>
> Date: Fri, 23 Sep 2011 17:32:08 -0700

I said everything I could in this matter.  What is left is discussion
of what is limbo and what the church thinks about that, for which I
have neither time nor motivation.

Just a few minor remarks, before I get back to fixing the bug I was
working on before this erupted.

> I mention Richard here because he is more likely than I to listen to and
> understand correctly the explanations about the design.

We will not see what Richard thinks until we are done with basics.
Any serious discussion of these issues and the related trade-offs must
be based on good understanding of the details of the design and
implementation.  We are not there yet, sadly.  ("Sadly" because I wish
I had someone more experienced and talented than myself available to
delve into the design and talk about this during the past 2 years,
while I labored on it.)

> > Do you understand what kind and amount of highly technical stuff
> > will have to be in the manual in order to explain this in enough
> > detail for users to understand it?
> 
> What kind and amount of "highly technical stuff" did you have to go into, to
> communicate to us those few corner cases?

They are not "few".  I just mentioned a few, but that doesn't mean
that's all there is to it.

> Then so is the same variable as a NON-option bogus.  And then so is its
> treatment in two (count 'em!) Emacs manuals.  I didn't create that variable, nor
> did I describe it in the manuals.

There's a very good reason why this variable is in the manuals.  The
words used to describe it in both manuals were carefully chosen, both
to avoid any factual inaccuracies and still leave it vague enough to
allow changes in the underlying implementation.  In particular, it's
not an incident that it expressly does NOT say in any of the 2 manuals
what is the precise effect on reordering of buffer text if this
variable is set to a nil value.

I encourage you to re-read the text with that in mind.  Maybe then you
will understand why I described it, even though using it means living
closer to the edge, and why the fact that it is documented says
nothing at all about the legitimacy of using it as a user-level
toggle.

bug#9571: 24.0.50; user option to turn off bidi, please

[Juanma Barranquero]

On Sat, Sep 24, 2011 at 05:46, Drew Adams <drew.adams@oracle.com> wrote:

> Thinking of you in that regard was secondary, and only because, as I say, you
> are one who is both knowledgable and wont to correct inaccuracies even in
> non-technical matters.  And yes, you can properly take that as a compliment,
> from someone who is also sometimes interested in non-software topics.

I wouldn't dream of taking it otherwise.

    Juanma

bug#9571: 24.0.50; user option to turn off bidi, please

[Juanma Barranquero]

On Sat, Sep 24, 2011 at 08:49, Eli Zaretskii <eliz@gnu.org> wrote:

> I said everything I could in this matter.  What is left is discussion
> of what is limbo and what the church thinks about that, for which I
> have neither time nor motivation.

Still, I think theology is quite on topic for Emacs developers... ;-)

    Juanma

bug#9571: 24.0.50; user option to turn off bidi, please

[Richard Stallman]

    > Why are you opposed to a flag to turn bidi display off?

    I explained that at length in followups.

Those messages seem to be arguing against maintaining big changes to
implement non-bidi display.  I agree with that.

But they don't seem to be an argument against simple code that would
disable the recognition of the bidi specialness of characters.

    >                 If there are [no R2L characters], there is no
    > reordering, thus no possibility it can cause confusion.

    You are wrong: _all_ characters are displayed in Emacs 24 via the
    reordering engine.  It's just that plain left-to-right text emerges
    from that reordering in its original buffer order.  But the reordering
    engine doesn't "know" that, it just implements the rules of
    reordering.

We are miscommunicating.  When I say "no reordering", I'm not talking
about what code is running -- just how the text gets displayed.
When there are no R2L characters, the text will be displayed in L2R order,
which means no reordering in the display.

    That will not resolve any confusion.  As someone who happened to read
    R2L text in Emacs 23 (e.g., in email messages), I can assure you:
    seeing R2L text in buffer order confuses even more than seeing results
    of slightly incorrect reordering.  It makes the reading process very
    slow and error prone, even if your command of the language is very
    good.

That's an argument not to do your normal editing with such a mode.  I
only suggest we provide it as a way to check the order of characters
in the buffer, when needed.

    It's not useful for users, believe me.  It could be useful to someone
    who debugs Emacs display, but there's no need for user option for
    that use case.

I am not sure what the best user interface would be.
Perhaps a command to toggle the flag for the current buffer.

    Easier maintenance.  Emacs display engine is already more complex that
    humanly perceptible.  Having two divergent engines in one means
    unnecessarily complicating maintenance and slowing down development,

I agree with you, but I am not arguing for having two engines.
Only for having a flag.

See the other message for how I suggest implementing it.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use free telephony http://directory.fsf.org/category/tel/

bug#9571: 24.0.50; user option to turn off bidi, please

[Richard Stallman]

Would this change in bidi_get_type suffice to implement non-bidi
display?  (In addition, one needs to define the option
display-bidi-flag, etc.)

This is partly based on guessing, so maybe it is wrong.  (I don't know
to test it.)  But, if it is wrong, I think you would see in a second
what's wrong, and maybe you could make it right in another minute.

If it is really as easy as this, why say no?


=== modified file 'src/bidi.c'
*** src/bidi.c	2011-09-24 10:31:37 +0000
--- src/bidi.c	2011-09-24 10:34:31 +0000
***************
*** 115,120 ****
--- 115,123 ----
    if (default_type == UNKNOWN_BT)
      abort ();
  
+   if (NILP (Vdisplay_bidi_flag))
+     return default_type;
+ 
    if (override == NEUTRAL_DIR)
      return default_type;
  



-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use free telephony http://directory.fsf.org/category/tel/

bug#9571: 24.0.50; user option to turn off bidi, please

[Richard Stallman]

    If you can read the relevant scripts, then bidi is not going to confuse
    you. If you cannot read the scripts, then turning off bidi is not really
    going to help (probably find-file-literally would be a better option in
    this case).

For people who can read the scripts, simple ordinary use of bidi will
make complete sense.  Even for people who can't read them, simple
ordinary use will cause no confusion.  (I can't read Hebrew, but a
Hebrew word in an English text won't be any harder to cope with than
a few Chinese characters.)

But the bidi feature is complex, and complex cases might be confusing
to anyone.  Especially if the text is the result of some sort of
error, and doesn't really make sense at all.  These are the cases
where I think temporarily disabling reordering could be useful.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use free telephony http://directory.fsf.org/category/tel/

bug#9571: 24.0.50; user option to turn off bidi, please

[Eli Zaretskii]

> Date: Sat, 24 Sep 2011 08:28:20 -0400
> From: Richard Stallman <rms@gnu.org>
> CC: drew.adams@oracle.com, 9571@debbugs.gnu.org
> 
> Those messages seem to be arguing against maintaining big changes to
> implement non-bidi display.  I agree with that.
> 
> But they don't seem to be an argument against simple code that would
> disable the recognition of the bidi specialness of characters.

That's an entirely different request, it was never voiced before (and
I suspect that it won't satisfy people who argue for a user option to
disable reordering).  Assuming I understand the request correctly, see
below.

The way I understand this request is: make it so that the result of
reordering is the text in its original logical (i.e. buffer or string)
order.  The code that is involved in reordering, largely in bidi.c and
xdisp.c, will still work as it normally does.

Is this what you are asking for?  If so, then the way to have this is
to modify the relevant Unicode property of the R2L characters so that
they are treated as L2R.  This is easier in Lisp than in C, because
the table where these properties are kept is just a specialized form
of a char-table, and Lisp code can modify it.

Note that we currently have no mechanism for making this table of
Unicode properties buffer-local, so changing the properties will
affect all the windows on all the frames.  But since we are talking
about something that's supposed to be used for very short periods of
time for exploring rare problems, I think it's not too important.

If it's important, and Stefan and Chong agree, I can write a command
to do this.

Again, I don't think this is what the original proponents of the
option wanted, because implementing what you suggest will not bypass
any of the code in the new display.  IOW, I think it's an entirely
different feature.

bug#9571: 24.0.50; user option to turn off bidi, please

[Eli Zaretskii]

> Date: Sat, 24 Sep 2011 08:28:39 -0400
> From: Richard Stallman <rms@gnu.org>
> CC: stepnem@gmail.com, 9571@debbugs.gnu.org
> 
> Would this change in bidi_get_type suffice to implement non-bidi
> display?  (In addition, one needs to define the option
> display-bidi-flag, etc.)
> [...]
> +   if (NILP (Vdisplay_bidi_flag))
> +     return default_type;
> + 

No, this won't do what you want.  default_type comes from this code:

  default_type = (bidi_type_t) XINT (CHAR_TABLE_REF (bidi_type_table, ch));

This accesses the table of bidirectional properties and extracts from
there the bidirectional type of CH, the character we are interested
in.  The rest of the code deals with _overriding_ that type.  For any
R2L character, default_type is STRONG_R, and it will cause bidi.c to
reorder it into visual order.

What you want is to pretend that R2L characters get the type STRONG_L,
which is the type reserved for letters in left-to-right scripts.

But simply overwriting default_type with STRONG_L isn't right, either.
That's because as long as we run the code in bidi.c unaltered, there
are some characters whose bidirectional properties are still needed
for the code to work: newlines, paragraph separators, line separators,
and a few others.

So my suggestion to have what you want is different, see my other
mail.

> If it is really as easy as this, why say no?

I didn't say no to this suggestion, because it was never explicitly
requested, from my POV.  If you meant this from the beginning, then
I'm sorry for my misunderstanding of what you meant.

However, I'm quite sure I did understand the original request that
started this, and it wasn't this.

bug#9571: 24.0.50; user option to turn off bidi, please

[Richard Stallman]

    The way I understand this request is: make it so that the result of
    reordering is the text in its original logical (i.e. buffer or string)
    order.  The code that is involved in reordering, largely in bidi.c and
    xdisp.c, will still work as it normally does.

Exactly.

    Again, I don't think this is what the original proponents of the
    option wanted, because implementing what you suggest will not bypass
    any of the code in the new display.  IOW, I think it's an entirely
    different feature.

Maybe you're right.  I didn't follow the whole discussion.  I am not
sure why they particularly want to bypass code in the new display;
that seems like an internal implementation detail, and I don't see why
a user would care how the job gets done as long as the output is what
he wants.

    But simply overwriting default_type with STRONG_L isn't right, either.
    That's because as long as we run the code in bidi.c unaltered, there
    are some characters whose bidirectional properties are still needed
    for the code to work: newlines, paragraph separators, line separators,
    and a few others.

Ok, my proposed patch won't do the job.  But wouldn't a slightly more
complicated patch in the same place do the job?

The method of replacing the unicode property table seems to have
several drawbacks:

1. Creating the modified table is more work.

2. It is a big data structure, so having another one would be a waste.

3. It feels wrong to alter the information describing the characters.
This is a matter of choosing a different way to display some characters,
not a matter of redefining what they mean.

I think an easy implementation in bidi_get_type is to have an if
statement choose between the existing switch statement and a new
switch statement.  The new switch statement would return different
values that would avoid reordering and give the Emacs 23 behavior.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use free telephony http://directory.fsf.org/category/tel/

bug#9571: 24.0.50; user option to turn off bidi, please

[Eli Zaretskii]

> Date: Sat, 24 Sep 2011 13:30:41 -0400
> From: Richard Stallman <rms@gnu.org>
> CC: drew.adams@oracle.com, 9571@debbugs.gnu.org
> 
> 
> Ok, my proposed patch won't do the job.  But wouldn't a slightly more
> complicated patch in the same place do the job?

It could be done, but why hard-code in C what can be done in Lisp?
The advantage would be that more people will understand the code and
changing it does not require rebuilding Emacs.

> The method of replacing the unicode property table seems to have
> several drawbacks:
> 
> 1. Creating the modified table is more work.

I don't understand why: we have map-char-table to do that easily and
elegantly.

> 2. It is a big data structure, so having another one would be a waste.

I don't think modifying entries in a char-table creates a new one.  It
just modifies entries in the existing one.  Perhaps Handa-san could
comment on that.

> 3. It feels wrong to alter the information describing the characters.
> This is a matter of choosing a different way to display some characters,
> not a matter of redefining what they mean.

The information I propose to change is used only for reordering
characters into visual order.  I'm talking about the Bidi_class
attribute of the characters.  No other attribute needs to be changed.
And the change will be temporary; toggling the feature off will
restore the original types.

bug#9571: 24.0.50; user option to turn off bidi, please

[Richard Stallman]

    I don't think modifying entries in a char-table creates a new one.  It
    just modifies entries in the existing one.  Perhaps Handa-san could
    comment on that.

It would be terribly unclean to change that data structure globally
rather than make a copy.  And it would affect all buffers at once.
This ought to be a per-buffer feature.

    It could be done, but why hard-code in C what can be done in Lisp?

Because there is only one thing to be done, and it is so simple in C.

-- 
Dr Richard Stallman
President, Free Software Foundation
51 Franklin St
Boston MA 02110
USA
www.fsf.org  www.gnu.org
Skype: No way! That's nonfree (freedom-denying) software.
  Use free telephony http://directory.fsf.org/category/tel/

bug#9571: 24.0.50; user option to turn off bidi, please

[Glenn Morris]

It was explained that this won't be done. Closed as "wontfix".