`C-b' is backward-char, `left' is left-char - why?

`C-b' is backward-char, `left' is left-char - why?

[Drew Adams]

I'm curious. Why is it a good idea that `C-b' and `left' are no longer bound to
the same command?

I'm not asking about the difference; I can see that from the doc strings.  I'm
wondering why we've broken their longstanding correspondence.

Lots of Emacs and Emacs Lisp does things based on which commands are used.  It's
sometimes not enough that two commands behave the same or similarly.  If they
are different commands then some code will likely not DTRT - some code will at
least not treat them the same.

Even if the bidi stuff specifies the same behavior for `backward-char' and
`left-char' whenever there is in fact no bidirectional stuff present, that does
not mean that Emacs will behave the same for these two keys, simply because they
are bound to different commands.  As a trivial example, if you have code that
remaps or uses `substitute-key-definition' with `backward-char', it will no
longer work for `left' as well as `C-b'.

Why not make bidi optional?  Why not have a minor mode for the bidi stuff, and
only bind keys such as `left' to commands that are specific to bidi when that
mode is turned on?  Why make such an invasive, top-level change to Emacs?

I understand that bidi is a great addition to Emacs and will be welcomed by
folks around the world.  I also realize that it is complex to implement.  But
some of us will rarely, if ever, use it.  A priori, it doesn't seem like a great
idea to be changing basic default key bindings.

I say "a priori" because I would like to understand why this is necessary or a
good idea.  I'm not claiming it's a bad idea and should be undone.  Chalk it up
mainly to surprise, if you like: I was surprised to see that `C-h w
backward-char' did not list `left' as well as `C-b'.

Re: `C-b' is backward-char, `left' is left-char - why?

[Pascal J. Bourguignon]

"Drew Adams" <drew.adams@oracle.com> writes:

> I'm curious. Why is it a good idea that `C-b' and `left' are no longer bound to
> the same command?

C-h k <left> gives:

<left> runs the command backward-char, which is an interactive
                        ^^^^^^^^^^^^^

C-h k C-b gives:

C-b runs the command backward-char, which is an interactive built-in
                     ^^^^^^^^^^^^^

I don't see any difference in my emacs-version "23.2.1".



> I'm not asking about the difference; I can see that from the doc strings.  I'm
> wondering why we've broken their longstanding correspondence.

Major or minor modes however may set their own bindings.  Perhaps you
have a buffer in a mode that defines a command named left, and which
binds <left> to it?

You can check the binding of the current modes with C-h m

-- 
__Pascal Bourguignon__                     http://www.informatimago.com/
A bad day in () is better than a good day in {}.

Re: `C-b' is backward-char, `left' is left-char - why?

[Eli Zaretskii]

> From: "Drew Adams" <drew.adams@oracle.com>
> Date: Fri, 27 May 2011 13:40:37 -0700
> 
> I'm curious. Why is it a good idea that `C-b' and `left' are no longer bound to
> the same command?

Because `left' and `right' behave differently depending on the
bidirectional context, whereas C-f and C-b are independent of it.

> I'm not asking about the difference; I can see that from the doc strings.  I'm
> wondering why we've broken their longstanding correspondence.

Because users of right-to-left scripts expect the current behavior of
the arrow keys.

> Lots of Emacs and Emacs Lisp does things based on which commands are used.  It's
> sometimes not enough that two commands behave the same or similarly.  If they
> are different commands then some code will likely not DTRT - some code will at
> least not treat them the same.

If you can suggest a way of catering to expectations of bidi users
without binding differently arrow and keyboard keys to cursor movement
commands, please do.

> Even if the bidi stuff specifies the same behavior for `backward-char' and
> `left-char' whenever there is in fact no bidirectional stuff present

It does.

> Why not make bidi optional?

It _is_ optional: you can set bidi-display-reordering to nil.  (Well,
actually it's nil now, but the plan is to make it t at some point
before Emacs 24 is released.)

> Why not have a minor mode for the bidi stuff

Bidi cannot be a minor mode, because bidi reordering for display
should happen automatically whenever there are right-to-left
characters in a buffer.  Minor modes don't work that way.

Besides, the rest of the world does bidi automatically; it's high time
Emacs does, too.

> only bind keys such as `left' to commands that are specific to bidi when that
> mode is turned on?  Why make such an invasive, top-level change to Emacs?

As I said: if you have practical suggestions (preferably with code),
let's hear them.  I made that change because every other program out
there differentiates the functions bound to these keys, but of course
if there's a better way of doing that in Emacs, I don't have any
dogmas here.

> I understand that bidi is a great addition to Emacs and will be welcomed by
> folks around the world.  I also realize that it is complex to implement.  But
> some of us will rarely, if ever, use it.

Emacs should behave exactly the same as it does without bidi when the
text doesn't include any right-to-left characters.  Anything else is a
bug.  The only reason why the bidi-display-reordering flag will stay
is because unibyte buffers should not be reordered.

Re: `C-b' is backward-char, `left' is left-char - why?

[Eli Zaretskii]

> From: "Pascal J. Bourguignon" <pjb@informatimago.com>
> Date: Fri, 27 May 2011 22:48:52 +0200
> 
> I don't see any difference in my emacs-version "23.2.1".

Try Emacs 24.

Re: `C-b' is backward-char, `left' is left-char - why?

[Eli Zaretskii]

> Date: Sat, 28 May 2011 00:09:31 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> > From: "Drew Adams" <drew.adams@oracle.com>
> > Date: Fri, 27 May 2011 13:40:37 -0700
> > 
> > I'm curious. Why is it a good idea that `C-b' and `left' are no longer bound to
> > the same command?
> 
> Because `left' and `right' behave differently depending on the
> bidirectional context, whereas C-f and C-b are independent of it.

And another reason: if `left' sometimes moves _forward_ in the buffer,
binding it to a command called `backward-char' is a lie.

RE: `C-b' is backward-char, `left' is left-char - why?

[Drew Adams]

> I don't see any difference in my emacs-version "23.2.1".

Yes, but in Emacs 24 things are different.
That's what I was referring to.

RE: `C-b' is backward-char, `left' is left-char - why?

[Drew Adams]

Thanks for the explanation.  Essentially, you've said that (a) the world wants
bidi and already uses it - which I already acknowledged, and (b) its redefining
of default keys _cannot_ be made optional because it needs to do certain things
automatically.

With my still limited understanding, I do not see the latter requirement, and I
do not see that you have demonstrated (explained) it.  See below.  That
Enacs-for-bidi needs to do certain things is one thing.  That it needs to be
imposed on everyone all the time, in order to be able to do those things when
appropriate is not so clear - not in your explanation so far.

> Because users of right-to-left scripts expect the current behavior of
> the arrow keys.
>
> > Why not make bidi optional?
> 
> It _is_ optional: you can set bidi-display-reordering to nil.

Obviously I meant optional in the sense that I was questioning.  How to prevent
it from binding default keys in this way?  You say it must bind `left' to
`left-char' instead of `backward-char'.  That obligation does not sound very
optional.

I will rarely, if ever, use bidi.  OK, maybe I'm a minority of one.  Still, how
can I prevent it from hijacking keys that normally would be bound to the same
good-ol' commands?  It doesn't help me much that the commands they are now bound
to might do the same thing, as I pointed out.  Commands are soemtimes used and
manipulated by code, in addition to being invoked interactively and individually
by users.  That's why we have things such as `remap' and
`substitute-key-definition'.

> > Why not have a minor mode for the bidi stuff
> 
> Bidi cannot be a minor mode, because bidi reordering for display
> should happen automatically whenever there are right-to-left
> characters in a buffer.  Minor modes don't work that way.

That it should automatically do things for people who want it I can understand.
But for someone who never wants it?

More importantly, what prevents a minor mode, which could even be enabled by
default for all I care, from doing just what you said: automatically...?  Turn
on the mode and you get what you've provided - everything.  Turn it off and you
get what Emacs has provided for years, limited as it might be.

I don't see how using a minor mode would prevent you from doing anything at all
that you want or need to do.  Can you please explain that?  At least having a
mode for this would give users a way to turn all of its effects off (hopefully -
at least the key hijacking I'm referring to).

IOW, if your `left-char' command makes sense only when bidi minor mode is turned
on, then it would not be made to hijack `left' except when that mode is on.
Minor modes can have their own keymaps, and if need be they can also be made to
change/restore other bindings (i.e. in different maps) when you turn them
on/off.

> Besides, the rest of the world does bidi automatically; it's high time
> Emacs does, too.

I have nothing against that.  I would like a way (if possible) for an individual
user to turn off its automatic sensitivity, rebinding of keys, etc.  That's all.
Even if there are few of us users who won't use bidi much, it would be nice to
give us the possibility of non-bidiness.

No one is trying to prevent bidi users from bidying (to coin a word).  Please do
not turn things around - I am not trying to prevent any bidiness among
consenting adults.  I'm all in favor of Emacs becoming bidi-enabled.  I only
hope that its bidification doesn't impose changes on people who don't need or
want to take advantage of bidi.

Enabling (or disabling) as an option - is that impossible?

> Emacs should behave exactly the same as it does without bidi when the
> text doesn't include any right-to-left characters.  Anything else is a
> bug.

See what I said in my first message.  Even if command `left-char' behaves the
same as command `backward-char' when the text doesn't include any..., that does
not help with code that remaps commands etc.

Many tests involving commands check only their symbols, not their definitions,
so even if these symbols had the _exact same_ function definition things would
break.  Think `substitute-key-definition'.

> And another reason: if `left' sometimes moves _forward_ in the buffer,
> binding it to a command called `backward-char' is a lie.

So create an alias.  This is really not the point.

It would be good for a user to optionally be able to have the traditional
behavior of having `C-b' and `left' bound by default to the _same_ command (same
symbol), whether it is `froblorph' or `backward-char' or `left-char'.

In addition, if possible it would be good for a user to optionally have that
same command be what it has been since the big bang: `backward-char'.  Any code
expecting that default binding or command name would then have a better chance
of behaving as expected.

If, as you suggest, but haven't yet backed up, you simply _cannot_ put this
stuff in a minor mode for some reason - even turning the mode on by default if
you want, then so be it.  I don't see that restriction yet, and would like to
hear the `why'.  But if it is truly a hard and fast limit for some reason, then
so be it.  On n'arrete pas le progres.

Re: `C-b' is backward-char, `left' is left-char - why?

[Antoine Levitt]

28/05/11 00:08, Drew Adams
>> > Why not have a minor mode for the bidi stuff
>> 
>> Bidi cannot be a minor mode, because bidi reordering for display
>> should happen automatically whenever there are right-to-left
>> characters in a buffer.  Minor modes don't work that way.
>
> That it should automatically do things for people who want it I can understand.
> But for someone who never wants it?
>
> More importantly, what prevents a minor mode, which could even be enabled by
> default for all I care, from doing just what you said: automatically...?  Turn
> on the mode and you get what you've provided - everything.  Turn it off and you
> get what Emacs has provided for years, limited as it might be.
>
> I don't see how using a minor mode would prevent you from doing anything at all
> that you want or need to do.  Can you please explain that?  At least having a
> mode for this would give users a way to turn all of its effects off (hopefully -
> at least the key hijacking I'm referring to).
>
> IOW, if your `left-char' command makes sense only when bidi minor mode is turned
> on, then it would not be made to hijack `left' except when that mode is on.
> Minor modes can have their own keymaps, and if need be they can also be made to
> change/restore other bindings (i.e. in different maps) when you turn them
> on/off.

I don't want to get into the heated exchanges (to put things kindly)
that seem to usually result from your posts, but why do you need "C-f"
and "right" to be the same thing? If it's for some generic code (for
instance, preview.el has the variable preview-auto-reveal that needs to
know that C-f and "right" both enter the preview), then it needs to
accomodate this use case for people who do use bidi, and if it's for
private code, surely you can rebind them (which takes just one more line
in your .emacs than turning off the minor mode).

So what's the point of changing stuff that works fine and adding a
trivial minor mode that is going to pollute our already crowded C-h m?

Re: `C-b' is backward-char, `left' is left-char - why?

[PJ Weisberg]

On Fri, May 27, 2011 at 3:08 PM, Drew Adams <drew.adams@oracle.com> wrote:

> Enabling (or disabling) as an option - is that impossible?

Certainly not.  You can set the option like so:

(global-set-key (kbd "<left>") 'backward-char)

This discussion would be more productive if you brought in an example
of a bug caused by this issue.

-PJ

RE: `C-b' is backward-char, `left' is left-char - why?

[Drew Adams]

> why do you need "C-f" and "right" to be the same thing?

I don't.  Why do we need them to be different, by default?

That's the question I posed (`C-b' and `left', actually).

Haven't seen an answer yet, except that bidi needs them to be different.  The
question then is why bidi's-need-for-this needs to become
Emacs's-need-in-general (all the time, everywhere, for everyone)?

As I said clearly several times, if it must, it must.  Really not a big deal.
Just asking whether and why it must.

> If it's for some generic code (for
> instance, preview.el has the variable preview-auto-reveal 
> that needs to
> know that C-f and "right" both enter the preview), then it needs to
> accomodate this use case for people who do use bidi, and if it's for
> private code, surely you can rebind them (which takes just 
> one more line
> in your .emacs than turning off the minor mode).

I mentioned `substitute-key-definition' and `remap'.

Of course it's not a giant perturbance for users to figure this out and DTRT to
accommodate the default key changes.

I was just asking whether and why it's necessary.  As I said at the beginning:
"I'm curious".

> So what's the point of changing stuff that works fine

Uh, just who is changing things here?  Are `C-b' and `left' bound to different
keys in the current Emacs release (23.3)?  Are they in any previous release?

Or are you proposing that we leave `left' bound to `backward-char' - i.e. no
change?

> and adding a trivial minor mode that is going to pollute
> our already crowded C-h m?

That `C-h m' is crowded is certainly true.

If users can get bidiless behavior (no default key changes) simply by toggling
an option, that will be great.  I'm sure Eli would welcome the patch for that,
if you have one.

I don't have a better idea than a minor mode, but I know _zero_ about bidi and
its implementation.  Better ideas are certainly welcome.  As you said, "what's
the point of changing stuff that works fine?"  If we must, we must.  But must
we?  Why?

RE: `C-b' is backward-char, `left' is left-char - why?

[Drew Adams]

> > Enabling (or disabling) as an option - is that impossible?
> 
> Certainly not.  You can set the option like so:
> (global-set-key (kbd "<left>") 'backward-char)

Not at all what I meant, as you no doubt know.

> This discussion would be more productive if you brought in an example
> of a bug caused by this issue.

No one mentioned a bug, or even an "issue".

I mentioned `substitute-key-definition' and command remapping as a use case.

I did not say that this is a big deal.  I said I was curious why we need to now
change these longstanding default key bindings.  That's all.  Still no answer,
but if no one else cares then it's OK that I not understand.

Re: `C-b' is backward-char, `left' is left-char - why?

[Nix]

On 27 May 2011, Pascal J. Bourguignon spake thusly:

> "Drew Adams" <drew.adams@oracle.com> writes:
>
>> I'm curious. Why is it a good idea that `C-b' and `left' are no longer bound to
>> the same command?
>
> C-h k <left> gives:
>
> <left> runs the command backward-char, which is an interactive
>                         ^^^^^^^^^^^^^
> C-h k C-b gives:
>
> C-b runs the command backward-char, which is an interactive built-in
>                      ^^^^^^^^^^^^^
> I don't see any difference in my emacs-version "23.2.1".

Yeah. This is a bidi thing, which means it's a 24.1 feature, currently
in bzr trunk only.

I'd say that the change makes sense, and if there's ever a time to do a
thing like this, it's in a major development branch. People *expect*
things to change between 23.x and 24.1, and this change is unlikely to
be very hard to compensate for -- certainly easier than the *other*
compensations required for the more complex third-party modes to look
nice and pretty in bidi environments.

Or so it seems to me.

-- 
NULL && (void)

Re: `C-b' is backward-char, `left' is left-char - why?

[PJ Weisberg]

On Fri, May 27, 2011 at 4:23 PM, Drew Adams <drew.adams@oracle.com> wrote:
>> > Enabling (or disabling) as an option - is that impossible?
>>
>> Certainly not.  You can set the option like so:
>> (global-set-key (kbd "<left>") 'backward-char)
>
> Not at all what I meant, as you no doubt know.

I know, but is it worse than "(setq backward-is-always-left t)"?

RE: `C-b' is backward-char, `left' is left-char - why?

[Drew Adams]

> >> > Enabling (or disabling) as an option - is that impossible?
> >>
> >> Certainly not.  You can set the option like so:
> >> (global-set-key (kbd "<left>") 'backward-char)
> >
> > Not at all what I meant, as you no doubt know.
> 
> I know, but is it worse than "(setq backward-is-always-left t)"?

Of course it is.  It affects only one command.  Presumably your option would
affect all or most of the commands that are directional.

The point was that the default bindings of various commands have come in sets,
and we are breaking up some of those sets.

Again, not the end of the world.
The question was whether this is necessary.

Besides, that's hardly the option I would opt for.
More like (setq bidify-emacs nil).
Or (setq bidi-hands-off-default-keys t).

Re: `C-b' is backward-char, `left' is left-char - why?

[Mohsen BANAN]

>>>>> On Fri, 27 May 2011 16:19:16 -0700, "Drew Adams" <drew.adams@oracle.com> said:

  >> why do you need "C-f" and "right" to be the same thing?
  Drew> I don't.  Why do we need them to be different, by default?

  Drew> That's the question I posed (`C-b' and `left', actually).

  Drew> Haven't seen an answer yet, except that bidi needs them to be different.  The
  Drew> question then is why bidi's-need-for-this needs to become
  Drew> Emacs's-need-in-general (all the time, everywhere, for everyone)?

  Drew> As I said clearly several times, if it must, it must.  Really not a big deal.
  Drew> Just asking whether and why it must.

...

  Drew> I don't have a better idea than a minor mode, but I know _zero_ about bidi and
  Drew> its implementation.  Better ideas are certainly welcome.  As you said, "what's
  Drew> the point of changing stuff that works fine?"  If we must, we must.  But must
  Drew> we?  Why?

If you were to view emacs as a gift from the
engineering profession to humanity, and if you
were to re-read Eli's previous note your questions
are answered.

Additionally, please let me present the
perspective of one who needs, cares-about and uses
bidi -- note that after Latin, Perso/Arabic script
is the most widely used character set family on
this planet.

Just like you, I fire up emacs with its default
settings, I go into Gnus and open a message.

That message could be in English/Globish or
Farsi/Arabic/Hebrew (فارسى/عربى) or mixed.

Then I try to reply and edit it in Farsi and use
<right> and <left> keys as they make sense in my
context. Why should that natural behavior not be
default? Why should I, as a bidi user, have to over
write any defaults to do what could be natural for
both you (non-bidi-user) and I (bidi-user). 
Are non-Latin character set users of emacs second
class citizens in your view?

Because left-to-right got done before
right-to-left, now <left> can not be different
from 'C-b'? 

Sure, there could be some lines of code related to
this that would need to change as we go from 23 to
24 (how big of a deal is that anyway?)

Please explain to us why you think that Eli's
solution is not the most reasonable? I am curious.

...محسن
...Mohsen

Re: `C-b' is backward-char, `left' is left-char - why?

[Stefan Monnier]

> Why not make bidi optional?  Why not have a minor mode for the bidi
> stuff, and only bind keys such as `left' to commands that are specific
> to bidi when that mode is turned on?  Why make such an invasive,
> top-level change to Emacs?

You mean that the `left' binding to `left-char' should be placed in
a minor mode keymap activated by the variable bidi-display-reordering?
[ I'll assume that's what you mean, if not, please tell us how you'd
  like to get the behavior you describe. ]

I can see why you'd want that, but there are some serious problems with it:

- minor mode bindings have higher precedence than major mode bindings, so
  suddenly major mode bindings of `left' would become ineffective.

- bidi-display-reordering is intended to default to t, so any problem
  that the above would avoid when bidi-display-reordering is nil would
  only affect a minority of users, while the majority would still be
  affected by those problems.


        Stefan

RE: `C-b' is backward-char, `left' is left-char - why?

[Drew Adams]

> note that after Latin, Perso/Arabic script
> is the most widely used character set family on
> this planet.

No one is arguing with you.

Personally, I'm ignorant of charset/script usage.  If I had to naively guess I
would have guessed that Chinese chars were more commonly used than Latin ones.

The point is that no one is making this into a contest between scripts or
charsets - except maybe you.

> Then I try to reply and edit it in Farsi and use
> <right> and <left> keys as they make sense in my
> context. Why should that natural behavior not be
> default? Why should I, as a bidi user, have to over
> write any defaults to do what could be natural for
> both you (non-bidi-user) and I (bidi-user). 
> Are non-Latin character set users of emacs second
> class citizens in your view?

Did you actually read anything I wrote?  Get off your high, first-class horse,
please.

Who said that what you describe should not be the default behavior?  I said
clearly that it makes sense for bidi behavior to _be_ the default.  This
particular pretty-much-latin-only user _wants_ bidi behavior to be the default.

What I asked about was the possibility of opting out of the key-binding changes
I described (no, not by creating separate individual bindings).

No one said you would or should need to opt in as a bidi user.  I suggested, as
a possible addition to the implementation, having a minor mode to somehow
control this.  And I made it clear that the default for that mode (or whatever)
could be bidi ON.  If I wasn't clear enough, let me say again that the default
not only could but _should_ be bidi ON, not off.

And if the only way to allow bidi support is to change longstanding default
bindings, then so be it.  I've said that too several times now.  My question was
whether, and if so why, that is necessary.  I would like to understand it a bit.

That's the only question I raised.  There never has been any question of not
having bidi support turned on by default.  No one is trying to interfere with
your ability to open Gnus and immediately have Farsi support.  Clear enough?

> Because left-to-right got done before
> right-to-left, now <left> can not be different
> from 'C-b'?

Read what I wrote.  Stop with the knee-jerk reactions, please.

> Please explain to us why you think that Eli's
> solution is not the most reasonable? I am curious.

I never claimed that it is not the most reasonable.  I have no idea whether it
is _the most_ reasonable.  Do you?  If I had to guess, I'd guess that it
probably is the most reasonable.  I know and respect Eli's ability to design and
implement.

That does not mean that he has always thought of everything from the outset, or
that everything he does is perfect.  He would probably be the first to
acknowledge that.  More importantly, it does not mean that no one can ask the
question why.

FWIW, I asked a similar question when Yidong changed key bindings for
delete-char (or whatever it was) and when similar changes were made for cursor
movement and visual-line mode (or whatever it is).  Likewise, for the recent
mouse selection changes.  In some cases people, including our best developers,
_have_ changed bindings and other settings unnecessarily, and this was later
backtracked (no, I don't recall specifics).

There is nothing wrong with asking the question whether we really need to change
these bindings globally and why.  Wouldn't you like to know?  _Especially_ if
there is a good reason and this is the most reasonable approach, I would like to
understand it.

You are overreacting.

RE: `C-b' is backward-char, `left' is left-char - why?

[Drew Adams]

> > Why not make bidi optional?  Why not have a minor mode for the bidi
> > stuff, and only bind keys such as `left' to commands that 
> > are specific to bidi when that mode is turned on?
> 
> You mean that the `left' binding to `left-char' should be placed in
> a minor mode keymap activated by the variable bidi-display-reordering?

I don't have a ready-made implementation, obviously.  Yes, I mentioned
minor-mode key bindings as one possibility.  But I also mentioned a minor mode
setting/restoring bindings in other keymaps.

The latter is problematic and not simple; it could be hairy and is not
necessarily clean.  And the former has obvious limitations, as you mention
below.

I was trying to suggest that it would be good to somehow be able to apply such a
set of bidi bindings only optionally (by default, but with the possibility of
opting out).

I did not try to provide a solution.  My point was to _ask_ whether changing
these bindings globally is necessary.  I simply asked whether there wasn't some
way to avoid changing them.  There's been a lot of heat, but not much in the way
of an explanation.  It's been said that they are needed for bidi.  It hasn't
been explained why they need to be in effect if bidi is disabled.

How the implementation would move between having these new bindings in place for
bidi and not making them when bidi is disabled is not something I would try to
answer.  I asked whether it was possible.

> I can see why you'd want that,

Well, good.  Is there a way to get it?  That's the question.

> - minor mode bindings have higher precedence than major mode 
>   bindings, so suddenly major mode bindings of `left' would
>   become ineffective.

Yes, that's a general problem with minor modes.  They are the closest thing I
can think of for what would be needed here to allow bidi to optionally affect
such bindings.  I recognize full well their limitations.

Re: `C-b' is backward-char, `left' is left-char - why?

[Mohsen BANAN]

Drew,

You are not paying attention to what is being said. 

I am not going to argue with you or repeat myself.

Everything I said directly related to default
behavior of <left> and <right> assuming bidi is
default. 

The problem is two fold. Your understanding of
bidi and your ability to listen.

You have been making a fuss over nothing in more
than one way.

...محسن
...Mohsen


>>>>> On Fri, 27 May 2011 18:53:26 -0700, "Drew Adams" <drew.adams@oracle.com> said:

  >> note that after Latin, Perso/Arabic script
  >> is the most widely used character set family on
  >> this planet.

  Drew> No one is arguing with you.

  Drew> Personally, I'm ignorant of charset/script usage.  If I had to naively guess I
  Drew> would have guessed that Chinese chars were more commonly used than Latin ones.

Chinese is not chars based.

  Drew> The point is that no one is making this into a contest between scripts or
  Drew> charsets - except maybe you.

  >> Then I try to reply and edit it in Farsi and use
  >> <right> and <left> keys as they make sense in my
  >> context. Why should that natural behavior not be
  >> default? Why should I, as a bidi user, have to over
  >> write any defaults to do what could be natural for
  >> both you (non-bidi-user) and I (bidi-user). 
  >> Are non-Latin character set users of emacs second
  >> class citizens in your view?

  Drew> Did you actually read anything I wrote?  Get off your high, first-class horse,
  Drew> please.

  Drew> Who said that what you describe should not be the default behavior?  I said
  Drew> clearly that it makes sense for bidi behavior to _be_ the default.  This
  Drew> particular pretty-much-latin-only user _wants_ bidi behavior to be the default.

  Drew> What I asked about was the possibility of opting out of the key-binding changes
  Drew> I described (no, not by creating separate individual bindings).

  Drew> No one said you would or should need to opt in as a bidi user.  I suggested, as
  Drew> a possible addition to the implementation, having a minor mode to somehow
  Drew> control this.  And I made it clear that the default for that mode (or whatever)
  Drew> could be bidi ON.  If I wasn't clear enough, let me say again that the default
  Drew> not only could but _should_ be bidi ON, not off.

  Drew> And if the only way to allow bidi support is to change longstanding default
  Drew> bindings, then so be it.  I've said that too several times now.  My question was
  Drew> whether, and if so why, that is necessary.  I would like to understand it a bit.

  Drew> That's the only question I raised.  There never has been any question of not
  Drew> having bidi support turned on by default.  No one is trying to interfere with
  Drew> your ability to open Gnus and immediately have Farsi support.  Clear enough?

  >> Because left-to-right got done before
  >> right-to-left, now <left> can not be different
  >> from 'C-b'?

  Drew> Read what I wrote.  Stop with the knee-jerk reactions, please.

  >> Please explain to us why you think that Eli's
  >> solution is not the most reasonable? I am curious.

  Drew> I never claimed that it is not the most reasonable.  I have no idea whether it
  Drew> is _the most_ reasonable.  Do you?  If I had to guess, I'd guess that it
  Drew> probably is the most reasonable.  I know and respect Eli's ability to design and
  Drew> implement.

  Drew> That does not mean that he has always thought of everything from the outset, or
  Drew> that everything he does is perfect.  He would probably be the first to
  Drew> acknowledge that.  More importantly, it does not mean that no one can ask the
  Drew> question why.

  Drew> FWIW, I asked a similar question when Yidong changed key bindings for
  Drew> delete-char (or whatever it was) and when similar changes were made for cursor
  Drew> movement and visual-line mode (or whatever it is).  Likewise, for the recent
  Drew> mouse selection changes.  In some cases people, including our best developers,
  Drew> _have_ changed bindings and other settings unnecessarily, and this was later
  Drew> backtracked (no, I don't recall specifics).

  Drew> There is nothing wrong with asking the question whether we really need to change
  Drew> these bindings globally and why.  Wouldn't you like to know?  _Especially_ if
  Drew> there is a good reason and this is the most reasonable approach, I would like to
  Drew> understand it.

  Drew> You are overreacting.

Re: `C-b' is backward-char, `left' is left-char - why?

[David Kastrup]

"Drew Adams" <drew.adams@oracle.com> writes:

>> >> > Enabling (or disabling) as an option - is that impossible?
>> >>
>> >> Certainly not.  You can set the option like so:
>> >> (global-set-key (kbd "<left>") 'backward-char)
>> >
>> > Not at all what I meant, as you no doubt know.
>> 
>> I know, but is it worse than "(setq backward-is-always-left t)"?
>
> Of course it is.  It affects only one command.  Presumably your option
> would affect all or most of the commands that are directional.

So you get the keybinding "backward-char" for C-b, and the keybinding
"maybe-backward-char-if-backward-is-always-left-is-set-and-left-char-otherwise"
for left.

Namely two different bindings, one looking up backward-is-always-left,
one ignoring it.

What is that supposed to buy you over backward-char and left-char?

> The point was that the default bindings of various commands have come
> in sets, and we are breaking up some of those sets.

So it appears that you want to have left-char and right-char, after
picking their choice of forward-char and backward-char, to check if any
of the latter have a remapping in place.

> The question was whether this is necessary.

That question has been answered.  That you refuse to hear the answer
does not change that.

-- 
David Kastrup

Re: `C-b' is backward-char, `left' is left-char - why?

[David Kastrup]

"Drew Adams" <drew.adams@oracle.com> writes:

> I did not try to provide a solution.  My point was to _ask_ whether
> changing these bindings globally is necessary.

Yes.

> I simply asked whether there wasn't some way to avoid changing them.

No.

> There's been a lot of heat, but not much in the way of an explanation.

You simply don't listen.

> It's been said that they are needed for bidi.  It hasn't been
> explained why they need to be in effect if bidi is disabled.

Because if they are supposed to be different when bidi is switched on,
that makes them different.  You don't gain anything by replacing
left-char with maybe-wrongly-backward-char-if-user-is-Drew.  You still
need something different from backward-char.

> How the implementation would move between having these new bindings in
> place for bidi and not making them when bidi is disabled is not
> something I would try to answer.  I asked whether it was possible.

Differing behavior is done by different keybindings.  That's where we
are.  If you worry about remapping of backward/forward-char, one could
make left-char/right-char look up the respective remap.  So far I have
seen no indication that there is any actual problem (instead of
hypothetical problems) solved by this.  So this complication does not
seem worth the trouble.  Worse, it will cause people not to fix stuff
that will break under bidi.

> Well, good.  Is there a way to get it?  That's the question.

The answer is no.  Not reasonably.  I have little doubt that you will
not accept this answer and prolong this thread as long as anybody cares
to react and then some.

-- 
David Kastrup

Re: `C-b' is backward-char, `left' is left-char - why?

[Eli Zaretskii]

> From: "Drew Adams" <drew.adams@oracle.com>
> Date: Fri, 27 May 2011 16:19:16 -0700
> 
> > why do you need "C-f" and "right" to be the same thing?
> 
> I don't.  Why do we need them to be different, by default?
> 
> That's the question I posed (`C-b' and `left', actually).

Btw, the same "problem" exists with `M-f'/`M-b' and
`C-<right>'/`C-<left>' as well.  We even discussed the possibility to
make `<Home>' and `<End>' behaving differently, see
https://lists.gnu.org/archive/html/emacs-devel/2010-06/msg00163.html
and the responses.

> Haven't seen an answer yet, except that bidi needs them to be different.

A simple answer is that I didn't see any other solution that is
simpler or more elegant than this one.  If you can suggest one, please
do.  Others explained (and I concur) that a minor mode is neither
simpler nor more elegant (more about that below).  If you (or someone
else, for that matter) have better suggestions, let's hear them.

> The question then is why bidi's-need-for-this needs to become
> Emacs's-need-in-general (all the time, everywhere, for everyone)?

Because experience shows that making such deep, central features
optional means trouble in more than one way.  For starters -- and this
was also mentioned in this thread -- it will make the bidi features
less reliable because many active contributors will opt not to use it
(e.g., because disabling it makes redisplay faster).

We've been through that once, when support for multibyte characters,
a.k.a. MULE, was introduced in Emacs 20.  I was there when it
happened.  If someone wants to put the Emacs community through this
again, I certainly don't, and I'm happy that Stefan and Chong agree.
Given this, you will understand that it is no incident that
bidi-display-reordering is not a user option.  If it's up to me, it
never will be.

Another reason is that once Emacs moved to be based on Unicode, it
needs to strive to support the requirements of the Unicode Standard as
much as possible, because users of word-processing programs come more
and more to expect that.  We already support various parts of that
standard, and we do it by default without any options and knobs.  The
Unicode Bidirectional Algorithm is part of that; that its support
needs significant changes in the Emacs infrastructure is a minor
detail, from the user perspective.  If OpenOffice and Firefox support
it by default (and I don't think you can disable it there), users will
expect Emacs to do the same.

> I mentioned `substitute-key-definition' and `remap'.

These are indeed valid considerations, but a minor mode is not the
solution, because the same problems will exist when that minor mode is
in effect.  Surely, you don't want to suggest that we should forget
about bidi users when we consider the difficulties with `remap' and
`substitute-key-definition'?

So if we don't want these consequences, we should come up with a
solution that works when bidi is in effect as well.  If you can
suggest such a solution, please do, but a minor mode isn't it.

Re: `C-b' is backward-char, `left' is left-char - why?

[Eli Zaretskii]

> From: "Drew Adams" <drew.adams@oracle.com>
> Cc: <emacs-devel@gnu.org>
> Date: Fri, 27 May 2011 15:08:46 -0700
> 
> > And another reason: if `left' sometimes moves _forward_ in the buffer,
> > binding it to a command called `backward-char' is a lie.
> 
> So create an alias.

An alias doesn't make the aliased function go away, it just gives it a
different name, for backward compatibility.  So let's talk about the
new name: are you saying that we should _rename_ `backward-char' into
something else? if so, what would that new name be?

The problem here is that C-f/C-b _always_ move forward/backward in the
buffer, while <right>/<left> _always_ move to the right resp. left
(well, _almost_ always, but let's forget about those complications for
a moment).  I think there's a need for both behaviors.  We could
perhaps find some way to make both sets of keys bound to the same
command, but how to call that command is not an insignificant issue.
I couldn't find a good name; can you?

> This is really not the point.

See, it _is_ the point, at least part of it.  The name of a command is
its best documentation, as you wrote many times here.  The name must
express what the command does, or users and Lisp programmers will be
confused.

> It would be good for a user to optionally be able to have the traditional
> behavior of having `C-b' and `left' bound by default to the _same_ command (same
> symbol), whether it is `froblorph' or `backward-char' or `left-char'.

I object to name that command `froblorph', but otherwise have no
serious objections to binding C-f/C-b and arrows to the same command,
provided that it will behave like right-char/left-char when invoked
through the arrow keys and like forward-char/backward-char when
invoked by C-f/C-b.  If you can show the code to do that, please do,
and let's then discuss whether your solution is better than what's
currently in the repository.

> In addition, if possible it would be good for a user to optionally have that
> same command be what it has been since the big bang: `backward-char'.

Not sure what you mean by "be what it has been": do you mean always to
move one character position back in the buffer?  If so, this is
already possible in at least 2 ways: (a) bind bidi-paragraph-direction
to left-to-right when invoking left-char, or (b) never use text that
includes characters from right-to-left scripts.

Re: `C-b' is backward-char, `left' is left-char - why?

[Eli Zaretskii]

> From: "Drew Adams" <drew.adams@oracle.com>
> Date: Fri, 27 May 2011 18:54:51 -0700
> Cc: emacs-devel@gnu.org
> 
> My point was to _ask_ whether changing these bindings globally is
> necessary.  I simply asked whether there wasn't some way to avoid
> changing them.  There's been a lot of heat, but not much in the way
> of an explanation.

Since the blame for this change is mine and mine alone (I did discuss
it with Stefan in private email, but only after making that change),
the only real explanation is what I already wrote in an earlier
message: I didn't find a better solution.

I happen to be happy with that solution, but I'm ready to hear
suggestions for better ones.

> How the implementation would move between having these new bindings in place for
> bidi and not making them when bidi is disabled is not something I would try to
> answer.  I asked whether it was possible.

In some complicated and inelegant way, sure.  But we here happen to
dislike complicated and inelegant ways when less complicated and more
elegant ones are available that have no significant disadvantages.

Re: `C-b' is backward-char, `left' is left-char - why?

[Stefan Monnier]

>> I can see why you'd want that,
> Well, good.  Is there a way to get it?  That's the question.

AFAICT, there is no way to get it in the general and default case (when
bidi-display-reordering is non-nil), which is the only case which would
make sense to solve.


        Stefan

Re: `C-b' is backward-char, `left' is left-char - why?

[Davis Herring]

On 05/29/2011 09:57 PM, Stefan Monnier wrote:
>>> I can see why you'd want that,
>> Well, good.  Is there a way to get it?  That's the question.
> 
> AFAICT, there is no way to get it in the general and default case (when
> bidi-display-reordering is non-nil), which is the only case which would
> make sense to solve.

I think what Drew wants is to have in startup.el something like

;; ...load .emacs
(if (default-value 'bidi-display-reordering)
    (global-set-key (kbd "<left>") 'left-char)
  (global-set-key (kbd "<left>") 'backward-char))
;; ...and C-f, M-b, M-f, etc.

along the lines of the other "preference detection" code like that for
unibyte.

I don't know if this is a good idea, but it does solve his problem
without introducing `magic-backward/left-dwim-char' bound to C-b and <left>.

Davis

-- 
This product is sold by volume, not by mass.  If it appears too dense or
too sparse, it is because mass-energy conversion has occurred during
shipping.

Re: `C-b' is backward-char, `left' is left-char - why?

[Eli Zaretskii]

> Date: Tue, 31 May 2011 08:18:56 -0600
> From: Davis Herring <herring@lanl.gov>
> Cc: Drew Adams <drew.adams@oracle.com>, emacs-devel@gnu.org
> 
> ;; ...load .emacs
> (if (default-value 'bidi-display-reordering)
>     (global-set-key (kbd "<left>") 'left-char)
>   (global-set-key (kbd "<left>") 'backward-char))
> ;; ...and C-f, M-b, M-f, etc.
> 
> along the lines of the other "preference detection" code like that for
> unibyte.
> 
> I don't know if this is a good idea, but it does solve his problem
> without introducing `magic-backward/left-dwim-char' bound to C-b and <left>.

A "good solution", if it exists, should solve the problem of 2
different commands not only for Drew, but for users of bidi as well.
If the problems raised by Drew are serious, they cannot be solved only
for some users, and should certainly be solved in the default
configuration.

Re: `C-b' is backward-char, `left' is left-char - why?

[Andy Moreton]

On Tue 31 May 2011, Davis Herring wrote:

> On 05/29/2011 09:57 PM, Stefan Monnier wrote:
>>>> I can see why you'd want that,
>>> Well, good.  Is there a way to get it?  That's the question.
>> 
>> AFAICT, there is no way to get it in the general and default case (when
>> bidi-display-reordering is non-nil), which is the only case which would
>> make sense to solve.
>
> I think what Drew wants is to have in startup.el something like
>
> ;; ...load .emacs
> (if (default-value 'bidi-display-reordering)
>     (global-set-key (kbd "<left>") 'left-char)
>   (global-set-key (kbd "<left>") 'backward-char))
> ;; ...and C-f, M-b, M-f, etc.

The help strings for 'left-char and 'backward-char could use some work.
It is unclear which moves according to screen display order and which
moves according to buffer character order.

    AndyM

Re: `C-b' is backward-char, `left' is left-char - why?

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Wed, 01 Jun 2011 12:48:15 +0100
> 
> The help strings for 'left-char and 'backward-char could use some work.
> It is unclear which moves according to screen display order and which
> moves according to buffer character order.

I don't feel a need to invest "some work" on the doc string of
`backward-char', because that function has not changed in ages,
certainly not now.  If its doc string is unclear, then I wonder how
did we all manage to use it all these years.

Of course, I don't object if someone wants to work on that doc string.

Regarding `left-char' and `right-char', the doc string says:

  Depending on the bidirectional context, this may move either
  backward or forward in the buffer.

Believe it or not, but I tried to make it more precise for a long
time, and this is the best I could come up with.  There's a slightly
different variant in the Emacs manual, maybe you will like it better.
But both are not 100% accurate, because explaining what exactly it
does would take a very long text that has no place in a doc string.

If, after playing with the function in bidirectional context, you have
suggestions for describing it better, please propose the change in the
doc string that you think would make it more clear.

Re: `C-b' is backward-char, `left' is left-char - why?

[Andy Moreton]

On Wed 01 Jun 2011, Eli Zaretskii wrote:

>> From: Andy Moreton <andrewjmoreton@gmail.com> Date: Wed, 01 Jun 2011
>> 12:48:15 +0100
>> 
>> The help strings for 'left-char and 'backward-char could use some
>> work. It is unclear which moves according to screen display order and
>> which moves according to buffer character order.
>
> I don't feel a need to invest "some work" on the doc string of
> `backward-char', because that function has not changed in ages,
> certainly not now. If its doc string is unclear, then I wonder how did
> we all manage to use it all these years.
>
> Of course, I don't object if someone wants to work on that doc string.

Having not looked at these functions previously, I found the doc strings
to be essentially the same for both of them, which was confusing.

> Regarding `left-char' and `right-char', the doc string says:
>
>   Depending on the bidirectional context, this may move either
>   backward or forward in the buffer.
>
> Believe it or not, but I tried to make it more precise for a long
> time, and this is the best I could come up with. There's a slightly
> different variant in the Emacs manual, maybe you will like it better.
> But both are not 100% accurate, because explaining what exactly it
> does would take a very long text that has no place in a doc string.

Having looked at the manual, I think the description there of <left> and
<right> does a better job of explaining what happens in bidi enabled
text than the doc strings. Given that a negative argument produces
movement in the opposite direction, I understand the difficulty of
explaining things.

> If, after playing with the function in bidirectional context, you have
> suggestions for describing it better, please propose the change in the
> doc string that you think would make it more clear.

If I understand things correctly the for N=1 (or omitted):

 - forward-char moves forward one character in the buffer, which in
   right-to-left text moves backward one position on the screen.

 - right-char moves forward one position on the screen, which in
   right-to-left text moves backward one character in the buffer.

 - backward-char moves backward one character in the buffer, which in
   right-to-left text moves forward one position on the screen.

 - left-char moves backward one position on the screen, which in
   right-to-left text moves forward one character in the buffer.

If so, is this attempt any better ?

--8<---------------cut here---------------start------------->8---
(forward-char &optional N)

Move point N characters forward (backward if N is negative).
On reaching end or beginning of buffer, stop and signal error.

Movement on the screen depends on the bidirectional context. Moving
forward over left-to-right text moves to the right on the screen, but
moving forward over right-to-left text moves _backward_ on the screen.
This is in contrast with right-char, which see.
--8<---------------cut here---------------end--------------->8---

    AndyM

Re: `C-b' is backward-char, `left' is left-char - why?

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Thu, 02 Jun 2011 00:26:59 +0100
> 
> Having not looked at these functions previously, I found the doc strings
> to be essentially the same for both of them, which was confusing.

I can understand the confusion, given the deceptively simple idea of
"forward" and "back" in the unidirectional world.

> Having looked at the manual, I think the description there of <left> and
> <right> does a better job of explaining what happens in bidi enabled
> text than the doc strings.

Well, we could duplicate the description from the manual in the doc
string, if deemed useful.

>  - forward-char moves forward one character in the buffer, which in
>    right-to-left text moves backward one position on the screen.

Not "backward", but to the left.  "Forward" and "backward" are not
well-defined on the screen in bidirectional context.  For someone who
reads the bidirectional text, C-f always moves forward, i.e. in the
reading direction, the direction in which we scan characters while
reading the text.

>  - right-char moves forward one position on the screen, which in
>    right-to-left text moves backward one character in the buffer.

This is misleading.  "Forward one position on the screen" is not well-
defined, so using it will confuse the reader.  Using "to the right"
instead is well-defined, but only _mostly_ correct, because there's no
pure right-to-left or left-to-right text.  Almost always there's some
left-to-right text (e.g., digits) embedded in predominantly
right-to-left text and vice versa.  (That's why the thing is called
"bidirectional".)  When you use right-char to move over left-to-right
characters embedded in otherwise right-to-left text, the cursor will
move to the left.  And that's even before we consider further
complications such as directional overrides (the RLE, RLO,
etc. control characters).

It would be more precise to tell that right-char will advance in the
buffer when used in a paragraph whose direction is left-to-right, but
will move backward in the buffer when used in a paragraph with
right-to-left direction.  But I'm not sure such a description will do
a better job, because explaining a simple idea such as cursor motion
via a much more complex issue of bidi-paragraph-direction doesn't
sound right.  By analogy, if you ask about laws wrt to traffic lights,
you will not be amused to first get a lecture about special relativity
theory that could potentially make the red light look green due to
Doppler effect.

> (forward-char &optional N)
> 
> Move point N characters forward (backward if N is negative).
> On reaching end or beginning of buffer, stop and signal error.
> 
> Movement on the screen depends on the bidirectional context. Moving
> forward over left-to-right text moves to the right on the screen, but
> moving forward over right-to-left text moves _backward_ on the screen.
                                               ^^^^^^^^^^^
Should be "_to_the_left_"

> This is in contrast with right-char, which see.

I'm okay with this if it helps to make the issue less vague, but you
should realize that it is also inaccurate, because "left-to-right
text" is not well-defined.  There are "weak" and "neutral" characters
whose directionality is defined by surrounding strong directional
characters.  There are directional override control characters that
can override the intrinsic properties of the following characters.
Etc., etc. -- all these can invalidate the simple description above.

Re: `C-b' is backward-char, `left' is left-char - why?

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andy Moreton <andrewjmoreton@gmail.com>
>> Date: Wed, 01 Jun 2011 12:48:15 +0100
>> 
>> The help strings for 'left-char and 'backward-char could use some work.
>> It is unclear which moves according to screen display order and which
>> moves according to buffer character order.
>
> I don't feel a need to invest "some work" on the doc string of
> `backward-char', because that function has not changed in ages,
> certainly not now.  If its doc string is unclear, then I wonder how
> did we all manage to use it all these years.
>
> Of course, I don't object if someone wants to work on that doc string.
>
> Regarding `left-char' and `right-char', the doc string says:
>
>   Depending on the bidirectional context, this may move either
>   backward or forward in the buffer.
>
> Believe it or not, but I tried to make it more precise for a long
> time, and this is the best I could come up with.

Correct me if I am wrong, but may it not also _jump_ backward or forward
in the buffer?  If I use right-char to move in an L2R context from L2R
text into a short embedded R2L text, I would expect it to move one
position to the right, namely _jump_ to the end of the R2L text, move
backwards over it till it comes to its beginning on the right, then
_jump_ to the following L2R text after its end on the left.

-- 
David Kastrup

Re: `C-b' is backward-char, `left' is left-char - why?

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Thu, 02 Jun 2011 09:23:37 +0200
> 
> > Regarding `left-char' and `right-char', the doc string says:
> >
> >   Depending on the bidirectional context, this may move either
> >   backward or forward in the buffer.
> >
> > Believe it or not, but I tried to make it more precise for a long
> > time, and this is the best I could come up with.
> 
> Correct me if I am wrong, but may it not also _jump_ backward or forward
> in the buffer?

You are not wrong, it certainly jumps on the L2R/R2L boundaries.  That
is why the doc string carefully avoids saying "move one character", it
just says "move", which I thought should include jumps.  Do you think
many people will interpret "move" as meaning just one glyph position
on the screen?

Re: `C-b' is backward-char, `left' is left-char - why?

[Andy Moreton]

On Thu 02 Jun 2011, Eli Zaretskii wrote:

>> From: Andy Moreton <andrewjmoreton@gmail.com>
>> Date: Thu, 02 Jun 2011 00:26:59 +0100
>
>>  - forward-char moves forward one character in the buffer, which in
>>    right-to-left text moves backward one position on the screen.
>
> Not "backward", but to the left.  "Forward" and "backward" are not
> well-defined on the screen in bidirectional context.  For someone who
> reads the bidirectional text, C-f always moves forward, i.e. in the
> reading direction, the direction in which we scan characters while
> reading the text.

Explaining motion in terms of buffer position (forward and backward),
reading direction (l2r/r2l) and screen position (left and right) may
help to disambiguate things as long as those terms are used consistently
and explained broefly in the doc string.

>>  - right-char moves forward one position on the screen, which in
>>    right-to-left text moves backward one character in the buffer.
>
> This is misleading.  "Forward one position on the screen" is not well-
> defined, so using it will confuse the reader.  Using "to the right"
> instead is well-defined, but only _mostly_ correct, because there's no
> pure right-to-left or left-to-right text.  Almost always there's some
> left-to-right text (e.g., digits) embedded in predominantly
> right-to-left text and vice versa.  (That's why the thing is called
> "bidirectional".)  When you use right-char to move over left-to-right
> characters embedded in otherwise right-to-left text, the cursor will
> move to the left.  And that's even before we consider further
> complications such as directional overrides (the RLE, RLO,
> etc. control characters).

I realised that this was a simplification, but it may make the overall
scheme clearer if the simple version is presented first and then the
additional complications explained.

> It would be more precise to tell that right-char will advance in the
> buffer when used in a paragraph whose direction is left-to-right, but
> will move backward in the buffer when used in a paragraph with
> right-to-left direction.  But I'm not sure such a description will do
> a better job, because explaining a simple idea such as cursor motion
> via a much more complex issue of bidi-paragraph-direction doesn't
> sound right.  By analogy, if you ask about laws wrt to traffic lights,
> you will not be amused to first get a lecture about special relativity
> theory that could potentially make the red light look green due to
> Doppler effect.

While overly complex descriptions are not useful, the doc strings do
need to convey whether the simple motion is relative to the buffer or
the screen position for each command.

>> (forward-char &optional N)
>> 
>> Move point N characters forward (backward if N is negative).
>> On reaching end or beginning of buffer, stop and signal error.
>> 
>> Movement on the screen depends on the bidirectional context. Moving
>> forward over left-to-right text moves to the right on the screen, but
>> moving forward over right-to-left text moves _backward_ on the screen.
>                                                ^^^^^^^^^^^
> Should be "_to_the_left_"
Agreed.

>> This is in contrast with right-char, which see.
>
> I'm okay with this if it helps to make the issue less vague, but you
> should realize that it is also inaccurate, because "left-to-right
> text" is not well-defined.  There are "weak" and "neutral" characters
> whose directionality is defined by surrounding strong directional
> characters.  There are directional override control characters that
> can override the intrinsic properties of the following characters.
> Etc., etc. -- all these can invalidate the simple description above.

I understand that there additional complications - I was trying to write
something that would explain the simpler cases so that the behaviour in
more complex cases could be inferred from the simpler ones.

So for forward-char, movement of point is always N characters forward in
the buffer. The effect on the screen position must be considered as a
sequence of single character movements in the buffer, each of which may
move the screen position left or right (depending on the bidirectional
context).

Is that more accurate ?

Thnaks for the patient explanations,

   AndyM

Re: `C-b' is backward-char, `left' is left-char - why?

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Thu, 02 Jun 2011 11:38:12 +0100
> 
> On Thu 02 Jun 2011, Eli Zaretskii wrote:
> 
> >> From: Andy Moreton <andrewjmoreton@gmail.com>
> >> Date: Thu, 02 Jun 2011 00:26:59 +0100
> >
> >>  - forward-char moves forward one character in the buffer, which in
> >>    right-to-left text moves backward one position on the screen.
> >
> > Not "backward", but to the left.  "Forward" and "backward" are not
> > well-defined on the screen in bidirectional context.  For someone who
> > reads the bidirectional text, C-f always moves forward, i.e. in the
> > reading direction, the direction in which we scan characters while
> > reading the text.
> 
> Explaining motion in terms of buffer position (forward and backward),
> reading direction (l2r/r2l) and screen position (left and right) may
> help to disambiguate things as long as those terms are used consistently
> and explained broefly in the doc string.

Yes.  We are in agreement.

> So for forward-char, movement of point is always N characters forward in
> the buffer. The effect on the screen position must be considered as a
> sequence of single character movements in the buffer, each of which may
> move the screen position left or right (depending on the bidirectional
> context).
> 
> Is that more accurate ?

Yes, this is accurate, though vague (because "bidirectional context"
is something left undefined).

Maybe the following variant of the 2nd sentence sounds better:

  The effect on the screen is to place the cursor on the character N
  buffer positions forward, which could be to the left or to the
  right, depending on the bidirectional context.

That's because Emacs doesn't really move point one character at a
time (when N is more than 1).

Re: `C-b' is backward-char, `left' is left-char - why?

[Andy Moreton]

On Thu 02 Jun 2011, Eli Zaretskii wrote:

>> From: Andy Moreton <andrewjmoreton@gmail.com>
>> Date: Thu, 02 Jun 2011 11:38:12 +0100
>> 
>> On Thu 02 Jun 2011, Eli Zaretskii wrote:
> Maybe the following variant of the 2nd sentence sounds better:
>
>   The effect on the screen is to place the cursor on the character N
>   buffer positions forward, which could be to the left or to the
>   right, depending on the bidirectional context.
>
> That's because Emacs doesn't really move point one character at a
> time (when N is more than 1).

I think that is definitely clearer than what we have now.

Similar wording is also needed for backward-char, left-char and
right-char as well. E.g. for right-char:

--8<---------------cut here---------------start------------->8---
(right-char &optional N)

Move point N characters to the right (to the left if N is negative). On
reaching beginning or end of buffer, stop and signal error.

The effect on the buffer is to place the cursor on the character N
screen positions to the right, which could be forward or backward from
the current position, depending on the bidirectional context.
--8<---------------cut here---------------end--------------->8---


    AndyM

Re: `C-b' is backward-char, `left' is left-char - why?

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Thu, 02 Jun 2011 13:59:48 +0100
> 
> > Maybe the following variant of the 2nd sentence sounds better:
> >
> >   The effect on the screen is to place the cursor on the character N
> >   buffer positions forward, which could be to the left or to the
> >   right, depending on the bidirectional context.
> >
> > That's because Emacs doesn't really move point one character at a
> > time (when N is more than 1).
> 
> I think that is definitely clearer than what we have now.

Thanks, I will make this change.

> (right-char &optional N)
> 
> Move point N characters to the right (to the left if N is negative). On
> reaching beginning or end of buffer, stop and signal error.
> 
> The effect on the buffer is to place the cursor on the character N
> screen positions to the right, which could be forward or backward from
> the current position, depending on the bidirectional context.

Here, as they say, the plot thickens: unlike C-f/C-b that _always_
move forward resp backward in the buffer, <right> and <left> don't
always move to the right resp to the left.  E.g., if you press <right>
in a paragraph whose bidi-paragraph-direction is left-to-right, then
the cursor will actually move to the _left_ when you get to some R2L
text embedded within this paragraph.  You can see an example of this
in etc/HELLO, in the lines that show Arabic and Hebrew welcome
phrases.

So if you invoke (right-char 10) when point is on characters from some
R2L script, the cursor could move to the left!

IOW, the names of <right> and <left> only express the _global_,
"grosso modo" direction of motion.  That generally DTRT (according to
user expectations) assuming that left-to-right paragraphs contain
mostly L2R text and only occasionally short sequences of R2L text; and
vice versa in right-to-left paragraphs.  But if a left-to-right
paragraph is made solely out of R2L text (a very rare and unusual
phenomenon), <right> will almost always move to the _left_, and <left>
to the right!  So in this case, even the large-scale movement is in
the "wrong" direction.

But while we could (for the doc string purposes) quite safely
disregard the use case of paragraph having the "wrong" direction and
disrupting the global movement direction as described above, the doc
string you suggest is wrong even locally, when short sequences of R2L
text are embedded in an otherwise left-to-right paragraph, or vice
versa.  This cannot be disregarded, so we must find a better way of
describing the effect of the arrow keys in mixed bidirectional text.
Ideas are welcome.

Re: `C-b' is backward-char, `left' is left-char - why?

[PJ Weisberg]

On Thursday, June 2, 2011, Andy Moreton <andrewjmoreton@gmail.com> wrote:

> the current position, depending on the bidirectional context.

A user familiar with only left-to-right languages might not know what
"bidirectional context" refers to here.

-- 

-PJ

Re: `C-b' is backward-char, `left' is left-char - why?

[Andy Moreton]

On Thu 02 Jun 2011, Eli Zaretskii wrote:

>> From: Andy Moreton <andrewjmoreton@gmail.com>
>> Date: Thu, 02 Jun 2011 13:59:48 +0100
>> 
>> > Maybe the following variant of the 2nd sentence sounds better:
>> >
>> >   The effect on the screen is to place the cursor on the character N
>> >   buffer positions forward, which could be to the left or to the
>> >   right, depending on the bidirectional context.
>> >
>> > That's because Emacs doesn't really move point one character at a
>> > time (when N is more than 1).
>> 
>> I think that is definitely clearer than what we have now.
>
> Thanks, I will make this change.
>
>> (right-char &optional N)
>> 
>> Move point N characters to the right (to the left if N is negative). On
>> reaching beginning or end of buffer, stop and signal error.
>> 
>> The effect on the buffer is to place the cursor on the character N
>> screen positions to the right, which could be forward or backward from
>> the current position, depending on the bidirectional context.
>
> Here, as they say, the plot thickens: unlike C-f/C-b that _always_
> move forward resp backward in the buffer, <right> and <left> don't
> always move to the right resp to the left.  E.g., if you press <right>
> in a paragraph whose bidi-paragraph-direction is left-to-right, then
> the cursor will actually move to the _left_ when you get to some R2L
> text embedded within this paragraph.  You can see an example of this
> in etc/HELLO, in the lines that show Arabic and Hebrew welcome
> phrases.
>
> So if you invoke (right-char 10) when point is on characters from some
> R2L script, the cursor could move to the left!

I find this to be baffling, but then I'm not the target audience for R2L
languages. Is this motion what users expect to happen for bidi text ?

So C-f/C-b move N characters in the buffer, then work out where that
lives on the screen (which may be to the right or left of the start
position. So far, so good.

> IOW, the names of <right> and <left> only express the _global_,
> "grosso modo" direction of motion.  That generally DTRT (according to
> user expectations) assuming that left-to-right paragraphs contain
> mostly L2R text and only occasionally short sequences of R2L text; and
> vice versa in right-to-left paragraphs.  But if a left-to-right
> paragraph is made solely out of R2L text (a very rare and unusual
> phenomenon), <right> will almost always move to the _left_, and <left>
> to the right!  So in this case, even the large-scale movement is in
> the "wrong" direction.

Now my head hurts :-)

> But while we could (for the doc string purposes) quite safely
> disregard the use case of paragraph having the "wrong" direction and
> disrupting the global movement direction as described above, the doc
> string you suggest is wrong even locally, when short sequences of R2L
> text are embedded in an otherwise left-to-right paragraph, or vice
> versa.  This cannot be disregarded, so we must find a better way of
> describing the effect of the arrow keys in mixed bidirectional text.
> Ideas are welcome.

OK, I completely misunderstood the semantics here, but your explanation
has again beeen enlightening. 

So is it that right-char means advance forward in screen display order
(which may move to the right or the left) and then work out which
buffer position it corresponds to ?

    AndyM

Re: `C-b' is backward-char, `left' is left-char - why?

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

> But while we could (for the doc string purposes) quite safely
> disregard the use case of paragraph having the "wrong" direction and
> disrupting the global movement direction as described above, the doc
> string you suggest is wrong even locally, when short sequences of R2L
> text are embedded in an otherwise left-to-right paragraph, or vice
> versa.  This cannot be disregarded, so we must find a better way of
> describing the effect of the arrow keys in mixed bidirectional text.
> Ideas are welcome.

It can even happen that both left and right move backward (when we are
at the end of an L2R piece in R2L context), or both move forward.

-- 
David Kastrup

Re: `C-b' is backward-char, `left' is left-char - why?

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Thu, 02 Jun 2011 17:23:01 +0100
> 
> > So if you invoke (right-char 10) when point is on characters from some
> > R2L script, the cursor could move to the left!
> 
> I find this to be baffling, but then I'm not the target audience for R2L
> languages. Is this motion what users expect to happen for bidi text ?

Yes, definitely.  Those expectations are the primary reason why I
wrote right-char and left-char (and also right-word and left-word),
and why I bound the arrow keys to them.

> So C-f/C-b move N characters in the buffer, then work out where that
> lives on the screen (which may be to the right or left of the start
> position.

Correct.  Btw, this has always been like that, even in the Emacs 23
unidirectional display.  Emacs never assumed that moving N characters
forward means the cursor should move N characters to the right on the
screen.  It always computed where to draw the cursor, it's just that
the unidirectional code which computed that assumed that character
positions increase linearly with screen positions.

> > IOW, the names of <right> and <left> only express the _global_,
> > "grosso modo" direction of motion.  That generally DTRT (according to
> > user expectations) assuming that left-to-right paragraphs contain
> > mostly L2R text and only occasionally short sequences of R2L text; and
> > vice versa in right-to-left paragraphs.  But if a left-to-right
> > paragraph is made solely out of R2L text (a very rare and unusual
> > phenomenon), <right> will almost always move to the _left_, and <left>
> > to the right!  So in this case, even the large-scale movement is in
> > the "wrong" direction.
> 
> Now my head hurts :-)

Well, maybe if you look at the body of right-char, you will see the
light:

  (if (eq (current-bidi-paragraph-direction) 'left-to-right)
      (forward-char n)
    (backward-char n)))

That's all there is to it: it does either forward-char or
backward-char, depending on the base direction of the current
paragraph.  And we've already established that forward-char and
backward-char can move to the left or to the right according to the
text across which they move.

The paragraph direction determines how the paragraph is displayed: in
a left-to-right paragraph, lines begin at the left margin of the
window, while in the right-to-left paragraph they begin at the right
margin.

> So is it that right-char means advance forward in screen display order
> (which may move to the right or the left) and then work out which
> buffer position it corresponds to ?

No, see the body of right-char above.  IOW, both forward-char and
right-char move in the buffer order, it's just that they sometimes
move in the same direction of buffer positions and sometimes in the
opposite ones.

(It is actually impossible to do what you say, because right-char must
be able to work even when the display is not up to date, e.g. if one
leans on the key and has a fast auto-repeat rate, or if some complex
Lisp program calls it.  By contrast, "screen display" is not defined
unless Emacs succeeded to catch up with user keystrokes, and brought
the display in sync with the buffer.)

Re: `C-b' is backward-char, `left' is left-char - why?

[Eli Zaretskii]

> Date: Thu, 2 Jun 2011 08:35:32 -0700
> From: PJ Weisberg <pjweisberg@gmail.com>
> Cc: "emacs-devel@gnu.org" <emacs-devel@gnu.org>
> 
> On Thursday, June 2, 2011, Andy Moreton <andrewjmoreton@gmail.com> wrote:
> 
> > the current position, depending on the bidirectional context.
> 
> A user familiar with only left-to-right languages might not know what
> "bidirectional context" refers to here.

Right.  Suggestions are welcome for how to say that less vaguely, but
without spilling the entire UAX#9 into the doc string ;-)

Re: `C-b' is backward-char, `left' is left-char - why?

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Thu, 02 Jun 2011 19:09:40 +0200
> 
> It can even happen that both left and right move backward (when we are
> at the end of an L2R piece in R2L context), or both move forward.

Unless I misunderstand you, no, this cannot happen.  <left> and
<right> always move in opposite directions in the buffer at the same
buffer position, because they invoke opposite buffer movement commands
(forward-char or backward-char) on identical conditions.  If <left>
invokes forward-char, then <right> will always invoke backward-char at
the same buffer position, or vice versa.

If I'm mistaken, please show me an example where what you say can
happen does happen.

Re: `C-b' is backward-char, `left' is left-char - why?

[PJ Weisberg]

On Thu, Jun 2, 2011 at 10:44 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Thu, 2 Jun 2011 08:35:32 -0700
>> From: PJ Weisberg <pjweisberg@gmail.com>
>> Cc: "emacs-devel@gnu.org" <emacs-devel@gnu.org>
>>
>> On Thursday, June 2, 2011, Andy Moreton <andrewjmoreton@gmail.com> wrote:
>>
>> > the current position, depending on the bidirectional context.
>>
>> A user familiar with only left-to-right languages might not know what
>> "bidirectional context" refers to here.
>
> Right.  Suggestions are welcome for how to say that less vaguely, but
> without spilling the entire UAX#9 into the doc string ;-)

Something like "...depending on the natural direction of text around
the cursor.  See UAX#9 regarding the display of text in languages that
read right-to-left."

-PJ

Re: `C-b' is backward-char, `left' is left-char - why?

[Eli Zaretskii]

> Date: Thu, 2 Jun 2011 12:29:04 -0700
> From: PJ Weisberg <pjweisberg@gmail.com>
> 
> On Thu, Jun 2, 2011 at 10:44 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> >> Date: Thu, 2 Jun 2011 08:35:32 -0700
> >> From: PJ Weisberg <pjweisberg@gmail.com>
> >> Cc: "emacs-devel@gnu.org" <emacs-devel@gnu.org>
> >>
> >> On Thursday, June 2, 2011, Andy Moreton <andrewjmoreton@gmail.com> wrote:
> >>
> >> > the current position, depending on the bidirectional context.
> >>
> >> A user familiar with only left-to-right languages might not know what
> >> "bidirectional context" refers to here.
> >
> > Right.  Suggestions are welcome for how to say that less vaguely, but
> > without spilling the entire UAX#9 into the doc string ;-)
> 
> Something like "...depending on the natural direction of text around
> the cursor.  See UAX#9 regarding the display of text in languages that
> read right-to-left."

Hm... "natural direction" is hardly clear.  How about "reading
direction" instead?

Anyway, this is from the doc string of right-char, and we didn't yet
figure out how to rephrase its main part.  When we do, we can revisit
this particular aspect.

Thanks.

Re: `C-b' is backward-char, `left' is left-char - why?

[Andy Moreton]

On Thu 02 Jun 2011, Eli Zaretskii wrote:

> Well, maybe if you look at the body of right-char, you will see the
> light:
>
>   (if (eq (current-bidi-paragraph-direction) 'left-to-right)
>       (forward-char n)
>     (backward-char n)))
>
> That's all there is to it: it does either forward-char or
> backward-char, depending on the base direction of the current
> paragraph.  And we've already established that forward-char and
> backward-char can move to the left or to the right according to the
> text across which they move.
>
> The paragraph direction determines how the paragraph is displayed: in
> a left-to-right paragraph, lines begin at the left margin of the
> window, while in the right-to-left paragraph they begin at the right
> margin.

What happens if N is large enough to cross into another paragraph with a
different value for current-bidi-paragraph-direction - is the resulting
motion surprising for users ?

Thanks for continuing my remedial education in the ways of bidi text :-)

    AndyM

Re: `C-b' is backward-char, `left' is left-char - why?

[Kenichi Handa]

In article <vz18vtk34dn.fsf@gmail.com>, Andy Moreton <andrewjmoreton@gmail.com> writes:

> So for forward-char, movement of point is always N characters forward in
> the buffer. The effect on the screen position must be considered as a
> sequence of single character movements in the buffer, each of which may
> move the screen position left or right (depending on the bidirectional
> context).

> Is that more accurate ?

I'm not sure we should make the docstring more complex by
mentioning about composition, but when forward-char is used
interactively (i.e. by typing C-f), the resulting buffer
position is changed more than one character if the character
at point is composed with the following few characters.  It
is very typical in South and South East Asian text, but
should occur in this Latin case too "À".

---
Kenichi Handa
handa@m17n.org

Re: `C-b' is backward-char, `left' is left-char - why?

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Thu, 02 Jun 2011 22:42:05 +0100
> 
> What happens if N is large enough to cross into another paragraph with a
> different value for current-bidi-paragraph-direction - is the resulting
> motion surprising for users ?

It doesn't surprise me, if that answers your question.

Emacs is about the only application that can invoke keyboard commands
with a repetition counter.  So there's no "prior art" here for users
to expect something different.

But if this comes up as an issue at some point, we can always modify
left-char and right-char to make their decisions once per move.
Although I think the result would surely surprise more, because under
the right (or maybe left ;-) circumstances you could then invoke
right-char with a non-zero argument, and wind up at the same place
where you started!

Re: `C-b' is backward-char, `left' is left-char - why?

[Eli Zaretskii]

> From: Kenichi Handa <handa@m17n.org>
> Date: Fri, 03 Jun 2011 09:47:59 +0900
> Cc: emacs-devel@gnu.org
> 
> In article <vz18vtk34dn.fsf@gmail.com>, Andy Moreton <andrewjmoreton@gmail.com> writes:
> 
> > So for forward-char, movement of point is always N characters forward in
> > the buffer. The effect on the screen position must be considered as a
> > sequence of single character movements in the buffer, each of which may
> > move the screen position left or right (depending on the bidirectional
> > context).
> 
> > Is that more accurate ?
> 
> I'm not sure we should make the docstring more complex by
> mentioning about composition, but when forward-char is used
> interactively (i.e. by typing C-f), the resulting buffer
> position is changed more than one character if the character
> at point is composed with the following few characters.

I know about this, but is that really part of forward-char?  I rather
thought that the main command loop advances point in these cases.  It
also does that when we enter a portion of text that is intangible,
e.g. covered by display strings or by invisible text property.  Aren't
composed characters handled the same way?  If not, where does
forward-char do what you describe?

Re: `C-b' is backward-char, `left' is left-char - why?

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>> Date: Thu, 02 Jun 2011 19:09:40 +0200
>> 
>> It can even happen that both left and right move backward (when we are
>> at the end of an L2R piece in R2L context), or both move forward.
>
> Unless I misunderstand you, no, this cannot happen.  <left> and
> <right> always move in opposite directions in the buffer at the same
> buffer position, because they invoke opposite buffer movement commands
> (forward-char or backward-char) on identical conditions.  If <left>
> invokes forward-char, then <right> will always invoke backward-char at
> the same buffer position, or vice versa.
>
> If I'm mistaken, please show me an example where what you say can
> happen does happen.

lllllRRRRRRlllll
     ^

If I move left, I jump backward over the RL text to the end of the LR
text.  If I move right, I move 1 character backward in the RL text.

Now that is what I would expect to happen.  However, not the _current_
direction decides whether to reverse left/right movement, but the
_paragraph_ direction.  If this is a LR paragraph (like it likely is),
left will move right in the RRRRRR section and vice versa.

This is what Hebrew writers expect?  I would have thought that it is
more natural to make the difference on the current direction, so that
right moves right visually.  It currently only does it when current bidi
direction and paragraph bidi direction concur, otherwise it moves in the
opposite direction.

-- 
David Kastrup

Re: `C-b' is backward-char, `left' is left-char - why?

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Fri, 03 Jun 2011 16:35:12 +0200
> 
> lllllRRRRRRlllll
>      ^

Is this the visual order (on the screen) or the logical order (in the
buffer)?

> If I move left, I jump backward over the RL text to the end of the LR
> text.  If I move right, I move 1 character backward in the RL text.

IIUC the example, the first sentence is true, the second is false.

> Now that is what I would expect to happen.  However, not the _current_
> direction decides whether to reverse left/right movement, but the
> _paragraph_ direction.

What do you mean by the "current direction"?

> If this is a LR paragraph (like it likely is), left will move right
> in the RRRRRR section and vice versa.

True.

> This is what Hebrew writers expect?

Yes.

Re: `C-b' is backward-char, `left' is left-char - why?

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>> Date: Fri, 03 Jun 2011 16:35:12 +0200
>> 
>> lllllRRRRRRlllll
>>      ^
>
> Is this the visual order (on the screen) or the logical order (in the
> buffer)?
>
>> If I move left, I jump backward over the RL text to the end of the LR
>> text.  If I move right, I move 1 character backward in the RL text.
>
> IIUC the example, the first sentence is true, the second is false.
>
>> Now that is what I would expect to happen.  However, not the _current_
>> direction decides whether to reverse left/right movement, but the
>> _paragraph_ direction.
>
> What do you mean by the "current direction"?

Reading direction at point (possibly split into reading direction to the
left of point's screen position, and reading direction to the right of
point's screen position).

>> If this is a LR paragraph (like it likely is), left will move right
>> in the RRRRRR section and vice versa.
>
> True.
>
>> This is what Hebrew writers expect?
>
> Yes.

Weird.

-- 
David Kastrup

Re: `C-b' is backward-char, `left' is left-char - why?

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Fri, 03 Jun 2011 17:14:54 +0200
> 
> > What do you mean by the "current direction"?
> 
> Reading direction at point (possibly split into reading direction to the
> left of point's screen position, and reading direction to the right of
> point's screen position).

I'm actually very happy this is not what is needed, because otherwise
we'd need to perform a large part of reordering for moving in the
buffer (because you cannot always trust the display to be up to date).

And that is even before we talk about the ambiguity (which you mention
above) on the L2R/R2L boundaries, which would need to be resolved by
some complicated features on the user level.  These are nicely avoided
by the current behavior.

> >> This is what Hebrew writers expect?
> >
> > Yes.
> 
> Weird.

The idea is that <left> moves forward when the paragraph direction is
L2R, and <right> moves forward in R2L paragraphs.  But they both move
in the reading (a.k.a. "logical") order, which in Emacs means in the
direction of increasing character positions.  Moving in strict visual
order (i.e. always left or right on the screen) is also possible, but
less desirable, because that's not the order in which people read the
text.

But what you suggest is neither visual nor logical order, so it seems
to be the worst of both worlds.  I, for one, have trouble predicting
where I will wind up, and need to think carefully before I give the
right answer.  That's not a good UI, IMO.

Re: `C-b' is backward-char, `left' is left-char - why?

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

> The idea is that <left> moves forward when the paragraph direction is
> L2R, and <right> moves forward in R2L paragraphs.  But they both move
> in the reading (a.k.a. "logical") order, which in Emacs means in the
> direction of increasing character positions.  Moving in strict visual
> order (i.e. always left or right on the screen) is also possible, but
> less desirable, because that's not the order in which people read the
> text.

Left/Right are for positioning the cursor.  They have nothing to do with
the order people read the text in.  I don't want my mouse to change
direction on RL text.

> But what you suggest is neither visual nor logical order,

Huh?  I suggest visual order.  Period.

> so it seems to be the worst of both worlds.

I have no idea what you are imagining here.

> I, for one, have trouble predicting where I will wind up, and need to
> think carefully before I give the right answer.  That's not a good UI,
> IMO.

Press left, end up at the next position to the left.  Press right, end
up at the next position to the right.  No need to even recognize whether
the glyphs at question are R2L or L2R: the effect is immediately
obvious.  Without thinking, without even knowing anything about the
R2L/L2R properties of the glyphs.  forward-character and
backward-character require more thinking.

-- 
David Kastrup

Re: `C-b' is backward-char, `left' is left-char - why?

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Fri, 03 Jun 2011 22:56:53 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > The idea is that <left> moves forward when the paragraph direction is
> > L2R, and <right> moves forward in R2L paragraphs.  But they both move
> > in the reading (a.k.a. "logical") order, which in Emacs means in the
> > direction of increasing character positions.  Moving in strict visual
> > order (i.e. always left or right on the screen) is also possible, but
> > less desirable, because that's not the order in which people read the
> > text.
> 
> Left/Right are for positioning the cursor.  They have nothing to do with
> the order people read the text in.

Evidently, they do.  At least many applications behave like Emacs
does.

> I don't want my mouse to change direction on RL text.

Not sure what "mouse direction" means here, and in what gestures.

> > But what you suggest is neither visual nor logical order,
> 
> Huh?  I suggest visual order.  Period.

My misunderstanding, sorry.

> forward-character and backward-character require more thinking.

Not if you actually read the text, because the movement is in the
reading order.

Re: `C-b' is backward-char, `left' is left-char - why?

[Kenichi Handa]

In article <83r57be6a9.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> > I'm not sure we should make the docstring more complex by
> > mentioning about composition, but when forward-char is used
> > interactively (i.e. by typing C-f), the resulting buffer
> > position is changed more than one character if the character
> > at point is composed with the following few characters.

> I know about this, but is that really part of forward-char?  I rather
> thought that the main command loop advances point in these cases.

Yes, that's why I wrote "when forward-char is used
interactively, the resulting buffer position is changed more
than one character".  If C-f results in movement of more
than one position, normal users think it's done by C-f.  I
just thought it may be good to have a pointer to a place
describing a position adjustment done by the command loop
(if any).

---
Kenichi Handa
handa@m17n.org

Re: `C-b' is backward-char, `left' is left-char - why?

[Eli Zaretskii]

> From: Kenichi Handa <handa@m17n.org>
> Cc: andrewjmoreton@gmail.com, emacs-devel@gnu.org
> Date: Sun, 05 Jun 2011 20:27:37 +0900
> 
> In article <83r57be6a9.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:
> 
> > > I'm not sure we should make the docstring more complex by
> > > mentioning about composition, but when forward-char is used
> > > interactively (i.e. by typing C-f), the resulting buffer
> > > position is changed more than one character if the character
> > > at point is composed with the following few characters.
> 
> > I know about this, but is that really part of forward-char?  I rather
> > thought that the main command loop advances point in these cases.
> 
> Yes, that's why I wrote "when forward-char is used
> interactively, the resulting buffer position is changed more
> than one character".  If C-f results in movement of more
> than one position, normal users think it's done by C-f.  I
> just thought it may be good to have a pointer to a place
> describing a position adjustment done by the command loop
> (if any).

OK, I will add something about that.

Re: `C-b' is backward-char, `left' is left-char - why?

[Ehud Karni]

On Fri, 03 Jun 2011 18:08:57 +0300, Eli Zaretskii wrote:
>
DK> If this is a LR paragraph (like it likely is), left will move right
DK> in the RRRRRR section and vice versa.
>
> True.
>
DK> This is what Hebrew writers expect?
>
> Yes.

I don't agree with you.  This is not what I expect, nor any bidi
novice, this is what Microsoft and openoffice has forced all the
bidi users to live with.

I expect a strict visual movement, i.e. the right arrow moves the
cursor to the right, left arrow moves to the left, no matter what
the language or the paragraph direction.

I think I know why Microsoft (and following them, openoffice) did
this non intuitive choice - They use shift+arrow to select text
strings and the string must be in adjacent memory locations.

Firefox (on Gnu/Linux and MS Windows) works in a strict visual manner
when the arrows are used without shift.  When shift+arrow is used (text
select) it behaves the same as openoffice (moves according to the
logical order).

It is interesting to check how KDE4, Gnome3 and Google Chrome are
behaving.


Ehud.


--
 Ehud Karni           Tel: +972-3-7966-561  /"\
 Mivtach - Simon      Fax: +972-3-7976-561  \ /  ASCII Ribbon Campaign
 Insurance agencies   (USA) voice mail and   X   Against   HTML   Mail
 http://www.mvs.co.il  FAX:  1-815-5509341  / \
 GnuPG: 98EA398D <http://www.keyserver.net/>    Better Safe Than Sorry

Re: `C-b' is backward-char, `left' is left-char - why?

[Eli Zaretskii]

> Date: Sun, 5 Jun 2011 19:51:10 +0300
> From: "Ehud Karni" <ehud@unix.mvs.co.il>
> Cc: dak@gnu.org, emacs-devel@gnu.org
> 
> On Fri, 03 Jun 2011 18:08:57 +0300, Eli Zaretskii wrote:
> >
> DK> If this is a LR paragraph (like it likely is), left will move right
> DK> in the RRRRRR section and vice versa.
> >
> > True.
> >
> DK> This is what Hebrew writers expect?
> >
> > Yes.
> 
> I don't agree with you.  This is not what I expect, nor any bidi
> novice, this is what Microsoft and openoffice has forced all the
> bidi users to live with.

Let me rephrase: that's what users expect because they were
brainwashed by MS and OpenOffice (and a few more apps that followed
suit).  Users who are brainwash-resistant want strict visual cursor
motion.

OK?

> I expect a strict visual movement, i.e. the right arrow moves the
> cursor to the right, left arrow moves to the left, no matter what
> the language or the paragraph direction.

It's possible to implement this as well, although a bit more tricky,
e.g. because it's not clear how to behave when the next glyph to the
right/left is on a display string or some such.  I also expect
difficulties in continuation lines and such likes.

But I still maintain that the decision to implement the logical
movement first was the right one, both because most users are of the
"brainwashed" variety, and because it is needed for various Emacs
features, such as shift-selections.

> I think I know why Microsoft (and following them, openoffice) did
> this non intuitive choice - They use shift+arrow to select text
> strings and the string must be in adjacent memory locations.

So does Emacs.

Re: `C-b' is backward-char, `left' is left-char - why?

[Ehud Karni]

On Sun, 05 Jun 2011 20:10:32 +0300, Eli Zaretskii wrote:
>
> > I think I know why Microsoft (and following them, openoffice) did
> > this non intuitive choice - They use shift+arrow to select text
> > strings and the string must be in adjacent memory locations.
>
> So does Emacs.

But Emacs does not select with the arrows, so it can work like Firefox
strict visual.

Ehud.


--
 Ehud Karni           Tel: +972-3-7966-561  /"\
 Mivtach - Simon      Fax: +972-3-7976-561  \ /  ASCII Ribbon Campaign
 Insurance agencies   (USA) voice mail and   X   Against   HTML   Mail
 http://www.mvs.co.il  FAX:  1-815-5509341  / \
 GnuPG: 98EA398D <http://www.keyserver.net/>    Better Safe Than Sorry

Re: `C-b' is backward-char, `left' is left-char - why?

[David Kastrup]

"Ehud Karni" <ehud@unix.mvs.co.il> writes:

> On Sun, 05 Jun 2011 20:10:32 +0300, Eli Zaretskii wrote:
>>
>> > I think I know why Microsoft (and following them, openoffice) did
>> > this non intuitive choice - They use shift+arrow to select text
>> > strings and the string must be in adjacent memory locations.
>>
>> So does Emacs.
>
> But Emacs does not select with the arrows, so it can work like Firefox
> strict visual.
shift-select-mode is a variable defined in `simple.el'.
Its value is t

Documentation:
When non-nil, shifted motion keys activate the mark momentarily.

While the mark is activated in this way, any shift-translated point
motion key extends the region, and if Transient Mark mode was off, it
is temporarily turned on.  Furthermore, the mark will be deactivated
by any subsequent point motion key that was not shift-translated, or
by any action that normally deactivates the mark in Transient Mark mode.

See `this-command-keys-shift-translated' for the meaning of
shift-translation.

You can customize this variable.

[back]


However, there is no reason that straight visual movement when using
shift-selection would interfere with selection as such: you just can't
expect that the marked region is visually contiguous.  The size of the
selection will jump when crossing visually from L2R and R2L.

But I see no logical problem with that.

-- 
David Kastrup

Re: `C-b' is backward-char, `left' is left-char - why?

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Sun, 05 Jun 2011 19:26:58 +0200
> 
> However, there is no reason that straight visual movement when using
> shift-selection would interfere with selection as such: you just can't
> expect that the marked region is visually contiguous.  The size of the
> selection will jump when crossing visually from L2R and R2L.
> 
> But I see no logical problem with that.

The problems with this are not logical, they are with implementing it
(both the movement itself and the resulting selection and highlight).
Patches are welcome.

Re: `C-b' is backward-char, `left' is left-char - why?

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>> Date: Sun, 05 Jun 2011 19:26:58 +0200
>> 
>> However, there is no reason that straight visual movement when using
>> shift-selection would interfere with selection as such: you just can't
>> expect that the marked region is visually contiguous.  The size of the
>> selection will jump when crossing visually from L2R and R2L.
>> 
>> But I see no logical problem with that.
>
> The problems with this are not logical, they are with implementing it
> (both the movement itself and the resulting selection and highlight).
> Patches are welcome.

The resulting selection and highlight appear to do just what is needed
already and don't seem to have a problem with visual discontinuity.

So only the movement itself would appear to be an issue.

If one has text like

llllllllRRRRRRRRllllll

there is a difference between the cursor being just to the right of the
first lll passage (namely before all of the RRR), and being just to the
left of the RRR passage (namely after all of the RRR).  Since being just
to the left of the RRR passage is the same point position as being just
to the left of the second lll passage, the effect of shift-marking while
moving left (let's reserve uppercase now for the marked passage) would
flip:

llllllllrrrrRrrrllllll
llllllllrrrRRrrrllllll
llllllllrrRRRrrrllllll
llllllllrRRRRrrrllllll
llllllllRRRRRrrrllllll
lllllllLrrrrrRRRllllll

Actually, I've just tried entering mixed L2R and R2L stuff with the
keyboard and bidi-display-reordering set, and I find it quite
distracting that the insertion point for "reversed" text (with regard to
the current paragraph direction) gets increasingly distant from the
cursor itself.

-- 
David Kastrup

Re: `C-b' is backward-char, `left' is left-char - why?

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Sun, 05 Jun 2011 20:26:01 +0200
> 
> > The problems with this are not logical, they are with implementing it
> > (both the movement itself and the resulting selection and highlight).
> > Patches are welcome.
> 
> The resulting selection and highlight appear to do just what is needed
> already and don't seem to have a problem with visual discontinuity.
> 
> So only the movement itself would appear to be an issue.

Only if you accept the way the region is currently highlighted, as
several disjoint stretches of text.  People who want strict visual
operations might want something else, like contiguous regions.

> If one has text like
> 
> llllllllRRRRRRRRllllll

That's the easy use case.  Try figuring out the more complicated ones
with embeddings and directional overrides.  UAX#9 allows up to 62
levels of nested embeddings, with or without directional overrides,
and Emacs supports them all.  Logical movement through that is clear,
but visual one... last time I tried my mind boiled.

> llllllllRRRRRrrrllllll
> lllllllLrrrrrRRRllllll

I find this flipping confusing and unexpected, but if someone wants
this, I don't mind.

> Actually, I've just tried entering mixed L2R and R2L stuff with the
> keyboard and bidi-display-reordering set, and I find it quite
> distracting that the insertion point for "reversed" text (with regard to
> the current paragraph direction) gets increasingly distant from the
> cursor itself.

<Shrug> I just did the minimal change in the original redisplay
design, whereby the cursor is placed on the character _after_ the
insertion point in the logical order.  Even that required a total
rewrite of the function that figures out where to draw the cursor.
Patches are welcome to rewrite it again so as to keep the cursor
closer to the insertion point.

Actually, TRT would be to split the cursor in two, because it plays
two different roles whose correct positions in this case do not
coincide.  That would need an even deeper surgery, including in
terminal-specific parts -- xterm.c, w32term.c, etc.  And what to do on
a TTY with its hardware cursor?  IOW, if people are queuing up to
improve the bidi display, there's enough work here for several
lifetimes.

Re: `C-b' is backward-char, `left' is left-char - why?

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>> Date: Sun, 05 Jun 2011 20:26:01 +0200
>> 
>> > The problems with this are not logical, they are with implementing it
>> > (both the movement itself and the resulting selection and highlight).
>> > Patches are welcome.
>> 
>> The resulting selection and highlight appear to do just what is needed
>> already and don't seem to have a problem with visual discontinuity.
>> 
>> So only the movement itself would appear to be an issue.
>
> Only if you accept the way the region is currently highlighted, as
> several disjoint stretches of text.  People who want strict visual
> operations might want something else, like contiguous regions.
>
>> If one has text like
>> 
>> llllllllRRRRRRRRllllll
>
> That's the easy use case.  Try figuring out the more complicated ones
> with embeddings and directional overrides.  UAX#9 allows up to 62
> levels of nested embeddings, with or without directional overrides,
> and Emacs supports them all.  Logical movement through that is clear,
> but visual one... last time I tried my mind boiled.

[...]

>> Actually, I've just tried entering mixed L2R and R2L stuff with the
>> keyboard and bidi-display-reordering set, and I find it quite
>> distracting that the insertion point for "reversed" text (with regard to
>> the current paragraph direction) gets increasingly distant from the
>> cursor itself.
>
> <Shrug> I just did the minimal change in the original redisplay
> design, whereby the cursor is placed on the character _after_ the
> insertion point in the logical order.  Even that required a total
> rewrite of the function that figures out where to draw the cursor.
> Patches are welcome to rewrite it again so as to keep the cursor
> closer to the insertion point.

I have thought about it, and I guess the key point is ambiguity of
cursor display.  The cursor is usually displayed just after the last
inserted character.  Which means to the left in R2L contexts, and to the
right in L2R contexts.  For a vertical cursor between characters, this
means that

LLLL^RRRR

is ambiguous: we are either at the end of the LLLL passage, or at the
end of the RRRR passage.  For a block or underline cursor, the ambiguity
gets worse:

LLL£RRRR

can mean that the cursor is next before last in the L2R passage, or last
in the R2L passage.

I think what is needed in this particular case is a virtual fill
character (perhaps the direction switch glyph, but displayed with a
different face?) at the insertion point when we are inserting in reverse
paragraph direction and the cursor would be displayed on a forward
paragraph character (including newline), not logically adjacent to the
insertion point.  Either that, or generally switch cursor type for
reverse direction insertion, so that one knows whether one has an L2R or
R2L facing cursor.

> Actually, TRT would be to split the cursor in two, because it plays
> two different roles whose correct positions in this case do not
> coincide.

They diverge only on borders between L2R and R2L, and a virtual padding
like described above gives the cursor a buffer that allows it to never
split.

> That would need an even deeper surgery, including in terminal-specific
> parts -- xterm.c, w32term.c, etc.  And what to do on a TTY with its
> hardware cursor?

The virtual padding approach should work on a tty since it requires only
one cursor to display.

-- 
David Kastrup

Re: `C-b' is backward-char, `left' is left-char - why?

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Tue, 07 Jun 2011 10:51:13 +0200
> 
> I have thought about it, and I guess the key point is ambiguity of
> cursor display.  The cursor is usually displayed just after the last
> inserted character.  Which means to the left in R2L contexts, and to the
> right in L2R contexts.  For a vertical cursor between characters, this
> means that
> 
> LLLL^RRRR
> 
> is ambiguous: we are either at the end of the LLLL passage, or at the
> end of the RRRR passage.

Right.  As I wrote, there are two places on the screen where we could
display the cursor in these cases, and TRT would be to display
something in each one of them.  It would also make sense to make the
whole range of characters between these two spots stand out in color
or something.

There's some discussion of these issues here:

  http://www-01.ibm.com/software/globalization/topics/bidiui/index.jsp

> I think what is needed in this particular case is a virtual fill
> character (perhaps the direction switch glyph, but displayed with a
> different face?) at the insertion point when we are inserting in reverse
> paragraph direction and the cursor would be displayed on a forward
> paragraph character (including newline), not logically adjacent to the
> insertion point.  Either that, or generally switch cursor type for
> reverse direction insertion, so that one knows whether one has an L2R or
> R2L facing cursor.

Sorry, I don't understand the details of your proposals.  It would
help if you show some pictures.  (What are "direction switch glyph"
and "forward paragraph character"? what "cursor type" do you want to
switch to? etc.)

> > Actually, TRT would be to split the cursor in two, because it plays
> > two different roles whose correct positions in this case do not
> > coincide.
> 
> They diverge only on borders between L2R and R2L, and a virtual padding
> like described above gives the cursor a buffer that allows it to never
> split.

IIUC, your "virtual padding" is just one possible implementation of a
split cursor.  But what does "gives the cursor a buffer" mean?

> > That would need an even deeper surgery, including in terminal-specific
> > parts -- xterm.c, w32term.c, etc.  And what to do on a TTY with its
> > hardware cursor?
> 
> The virtual padding approach should work on a tty since it requires only
> one cursor to display.

Maybe I'll agree once I understand what you propose.  Right now, I
cannot judge if everything you propose can work on a TTY.  In any
case, the redisplay interface for displaying the cursor would need to
change, because we will now compute 2 locations, not 1.