Performance

Performance

[Stefan Monnier]

It seems that functionality-wise, the bidi code is almost as good as it
was before bidi.  But performancewise, I experience some problems.
I recently started to compile without -DENABLE_CHECKING and other such
debugging code (tho still with -O0 and -g) and am seeing cases where
cursor motion is slow.  The problem is most notable in operations such
as C-e or C-n/C-p (tho those tend to be fast enough as long as I'm in
column 0).
Redisplay itself also seems to be noticeable slower at times.

This usually shows up after I open a few different buffers&frames (like
when I have 10 frames or more (each one showing a single file), where
most frames are sized 80x89).

With -DENABLE_CHECKING, -DXASSERTS, and -DFONTSET_DEBUG, the slowdown is
*very* noticeable, to the point of making Emacs sometimes painful
to use.
My machines range from Atom netbooks to core i7, so not "too old to
be relevant".


        Stefan

Re: Performance

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Sun, 06 Jun 2010 21:05:54 -0400
> 
> It seems that functionality-wise, the bidi code is almost as good as it
> was before bidi.

Thanks for testing it.

> But performancewise, I experience some problems.
> I recently started to compile without -DENABLE_CHECKING and other such
> debugging code (tho still with -O0 and -g) and am seeing cases where
> cursor motion is slow.  The problem is most notable in operations such
> as C-e or C-n/C-p (tho those tend to be fast enough as long as I'm in
> column 0).
> Redisplay itself also seems to be noticeable slower at times.

I'm a bit surprised, because my main machine is 5 years old, and I
don't see any noticeable slowdown.  But then I almost never have 10
frames (let alone more) open at the same time; usually all but 2 are
minimized or iconified.

> This usually shows up after I open a few different buffers&frames (like
> when I have 10 frames or more (each one showing a single file), where
> most frames are sized 80x89).

I will try that (when I'm back from traveling in a few days).

Granted, I didn't yet get to profiling and optimizing the code,
because (a) so many features need yet to get right before I get them
fast, and (b) the current code works reasonably fast for me, even when
I compile with -O0 and even in a buffer with text that really needs
reordering (which I understand is not your case).

From what you tell, it sounds like vertical cursor motion is the
problem; please try C-f and C-b (_not_ the arrow keys!) and tell if
they are reasonably fast or also slow.

Even if C-f and C-b are fast, I don't immediately see any special
reasons for slowdown with vertical motion, in a buffer with text that
doesn't need reordering, except the code in bidi.c.  One way of
testing this hypothesis would be to replace the code in bidi.c with
incrementing it->charpos and it->bytepos, like the unidirectional
iteration does.

Also, what kind of files are those?  If it's human-readable text (not
program sources or some such), are paragraphs in it long or short?  Do
you see any change in performance if you set bidi-paragraph-direction
to `left-to-right'?

Anyway, I'd love some help in this matter.  Getting the bidi code
faster is not a trivial job, but it does not require any knowledge of
bidirectional scripts, and I can help with understanding what the code
does and why.  Without someone stepping forward, I doubt that I could
get to seriously working on speeding up the code, what with merely 10
hours a week I have to work on Emacs.

arrow keys vs. C-f/b/n/p (was: Performance)

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

> From what you tell, it sounds like vertical cursor motion is the
> problem; please try C-f and C-b (_not_ the arrow keys!) and tell if
> they are reasonably fast or also slow.

Since we opened that can of worms already by making C-f/C-b do logical
motion as opposed to the arrow keys doing visual motion, I should think
it would make sense to disconnect C-n/C-p from the vertical arrow keys
as well and move them in a non-display-related manner in the "keep the
same C-f distance from the last LF" sense.  I have no good answer what
to do about tabs, but I am likely in a lot of company there.  Anyway:
likely the same with "C-a" "C-e" for logical movement as contrasted to
<home> and <end> for visual movement.

It may make sense to allow switching everything to logical manner, or
everything to visual manner, but the default setting of "Use visual
motion in all regards for off-keyboard special keys, and logical for
everything on-keyboard" seems to be a good match to the brain areas
people are expecting to engage in connection with certain kinds of
keybindings.

-- 
David Kastrup

Re: arrow keys vs. C-f/b/n/p (was: Performance)

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Mon, 07 Jun 2010 16:22:51 +0200
> 
> Since we opened that can of worms already by making C-f/C-b do logical
> motion as opposed to the arrow keys doing visual motion, I should think
> it would make sense to disconnect C-n/C-p from the vertical arrow keys
> as well and move them in a non-display-related manner in the "keep the
> same C-f distance from the last LF" sense.

I have no objection to this suggestion, although I'm not sure when I
personally would use such kind of a down/up-arrow command.

> likely the same with "C-a" "C-e" for logical movement as contrasted to
> <home> and <end> for visual movement.

To this, I object.  I made the left/right arrows and their
Ctrl-modified siblings move to the left resp. right, regardless of the
logical buffer order, because this is what users of bidirectional
scripts expect -- they are used to it in every other bidi-aware
application out there.  By contrast, <home> and <end> always move to
the beginning resp. end of the current line, they have no relation to
either left or right directions.

Re: arrow keys vs. C-f/b/n/p

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> Since we opened that can of worms already by making C-f/C-b do logical
>> motion as opposed to the arrow keys doing visual motion, I should think
>> it would make sense to disconnect C-n/C-p from the vertical arrow keys
>> as well and move them in a non-display-related manner in the "keep the
>> same C-f distance from the last LF" sense.
>
> I have no objection to this suggestion, although I'm not sure when I
> personally would use such kind of a down/up-arrow command.

Well, in keyboard macros doing work on columnar data you don't want to
have them move into wrapped lines.

>> likely the same with "C-a" "C-e" for logical movement as contrasted
>> to <home> and <end> for visual movement.
>
> To this, I object.  I made the left/right arrows and their
> Ctrl-modified siblings move to the left resp. right, regardless of the
> logical buffer order, because this is what users of bidirectional
> scripts expect -- they are used to it in every other bidi-aware
> application out there.  By contrast, <home> and <end> always move to
> the beginning resp. end of the current line,

The left border of the screen is not necessarily the next character
after the line feed.

> they have no relation to either left or right directions.

On my keyboard, the <home> character is inscribed with "Pos1".  That is
arguably associated with the leftmost column.

-- 
David Kastrup

Re: arrow keys vs. C-f/b/n/p

[Stefan Monnier]

> Since we opened that can of worms already by making C-f/C-b do logical
> motion as opposed to the arrow keys doing visual motion,

Actually, the arrow keys don't do "visual motion".  They just use
logical motion like C-f/C-b except that they switch direction depending
on the current paragraph's default direction (i.e. right-arrow will
still move left when in an R2L piece of text within an L2R paragraph, as
can be seen in the arabic text in the HELLO file).

> I should think it would make sense to disconnect C-n/C-p from the
> vertical arrow keys as well and move them in a non-display-related
> manner in the "keep the same C-f distance from the last LF" sense.
> I have no good answer what to do about tabs, but I am likely in a lot
> of company there.  Anyway: likely the same with "C-a" "C-e" for
> logical movement as contrasted to <home> and <end> for
> visual movement.

I have no intention to make any such change for now.


        Stefan

Re: Performance

[Stefan Monnier]

> I'm a bit surprised, because my main machine is 5 years old, and I
> don't see any noticeable slowdown.  But then I almost never have 10
> frames (let alone more) open at the same time; usually all but 2 are
> minimized or iconified.

I think the number of frames is a key part to trigger the slowdown
(e.g. I don't notice it in the Emacs instance that runs Gnus where
I usually have only 2 or maybe 3 frames only).

Sometimes I see a significant slowdown even when only 2 frames are
"open" (typically one 80x89 plus a separate single-line minibuffer-only
frame), but with a large number of iconified frames (like 20 or more).

> Granted, I didn't yet get to profiling and optimizing the code,
> because (a) so many features need yet to get right before I get them
> fast, and (b) the current code works reasonably fast for me, even when
> I compile with -O0 and even in a buffer with text that really needs
> reordering (which I understand is not your case).

I figured as much, but thought we may want to start thinking about
performance since it can take a while to track it down (and then fix
it).

> From what you tell, it sounds like vertical cursor motion is the
> problem; please try C-f and C-b (_not_ the arrow keys!) and tell if
> they are reasonably fast or also slow.

C-f and C-b also have trouble keeping up with the repeat-rate (30/s).

> Also, what kind of files are those?

Code files, Help buffers, ... Most of them fit within 80 columns.

> Anyway, I'd love some help in this matter.  Getting the bidi code
> faster is not a trivial job, but it does not require any knowledge of
> bidirectional scripts, and I can help with understanding what the code
> does and why.  Without someone stepping forward, I doubt that I could
> get to seriously working on speeding up the code, what with merely 10
> hours a week I have to work on Emacs.

Of course.  Actually, part of the reason I posted it is so as to see if
other people have experienced performance problems.  I'm not completely
sure the problems are linked to cursor motion, to tell you the truth
(e.g. it might be linked to my systematic use of reveal-mode).
So I think we need a lot more data points in order to figure out where
the problem comes from.


        Stefan

Re: arrow keys vs. C-f/b/n/p (was: Performance)

[Uday S Reddy]

Eli Zaretskii <eliz@gnu.org> writes:

>> likely the same with "C-a" "C-e" for logical movement as contrasted to
>> <home> and <end> for visual movement.
>
> To this, I object.  I made the left/right arrows and their
> Ctrl-modified siblings move to the left resp. right, regardless of the
> logical buffer order, because this is what users of bidirectional
> scripts expect -- they are used to it in every other bidi-aware
> application out there.  By contrast, <home> and <end> always move to
> the beginning resp. end of the current line, they have no relation to
> either left or right directions.

If there is going to be an entire suite of commands for doing visual
motion, then it makes sense to have a uniform setting of key bindings
for them.  Key bindings can always be changed by the users (famous
last words!).  

The issue with C-n has been that the meaning of the command was
changed, not just a key binding.  It would have been less of an issue
if it was just a key binding.

Cheers,
Uday

Re: arrow keys vs. C-f/b/n/p

[Juanma Barranquero]

On Mon, Jun 7, 2010 at 17:19, Stefan Monnier <monnier@iro.umontreal.ca> wrote:

> Actually, the arrow keys don't do "visual motion".  They just use
> logical motion like C-f/C-b except that they switch direction depending
> on the current paragraph's default direction (i.e. right-arrow will
> still move left when in an R2L piece of text within an L2R paragraph, as
> can be seen in the arabic text in the HELLO file).

Moving around that line, I see no difference between using <right> and
C-f. What am I doing wrong?

    Juanma

Re: arrow keys vs. C-f/b/n/p

[David Kastrup]

Juanma Barranquero <lekktu@gmail.com> writes:

> On Mon, Jun 7, 2010 at 17:19, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>
>> Actually, the arrow keys don't do "visual motion".  They just use
>> logical motion like C-f/C-b except that they switch direction depending
>> on the current paragraph's default direction (i.e. right-arrow will
>> still move left when in an R2L piece of text within an L2R paragraph, as
>> can be seen in the arabic text in the HELLO file).
>
> Moving around that line, I see no difference between using <right> and
> C-f. What am I doing wrong?

There is no R2L paragraph in that file.

-- 
David Kastrup

Re: arrow keys vs. C-f/b/n/p

[Juanma Barranquero]

On Mon, Jun 7, 2010 at 18:21, David Kastrup <dak@gnu.org> wrote:

> There is no R2L paragraph in that file.

What does this mean, then?

> (i.e. right-arrow will
> still move left when in an R2L piece of text within an L2R paragraph, as
> can be seen in the arabic text in the HELLO file).

    Juanma

Re: arrow keys vs. C-f/b/n/p

[Andreas Schwab]

Juanma Barranquero <lekktu@gmail.com> writes:

> On Mon, Jun 7, 2010 at 18:21, David Kastrup <dak@gnu.org> wrote:
>
>> There is no R2L paragraph in that file.
>
> What does this mean, then?

Try (setq bidi-paragraph-direction 'right-to-left).

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: arrow keys vs. C-f/b/n/p

[Juanma Barranquero]

On Mon, Jun 7, 2010 at 20:26, Andreas Schwab <schwab@linux-m68k.org> wrote:

> Try (setq bidi-paragraph-direction 'right-to-left).

OK, I see the difference. I still don't see how it applies to what
Stefan said ("right-arrow will still move left when in an R2L piece of
text within an L2R paragraph").

    Juanma

Re: arrow keys vs. C-f/b/n/p

[Stefan Monnier]

>> Try (setq bidi-paragraph-direction 'right-to-left).
> OK, I see the difference. I still don't see how it applies to what
> Stefan said ("right-arrow will still move left when in an R2L piece of
> text within an L2R paragraph").

Have you tried to follow the "as can be seen in the arabic text in the
HELLO file" part for an example?


        Stefan

Re: arrow keys vs. C-f/b/n/p

[Juanma Barranquero]

On Mon, Jun 7, 2010 at 21:57, Stefan Monnier <monnier@iro.umontreal.ca> wrote:

> Have you tried to follow the "as can be seen in the arabic text in the
> HELLO file" part for an example?

Yes. But I'm clearly misunderstanding something. With etc/HELLO as is,
if I move the cursor over the arabic line, both C-f and <right> do the
exact same thing. If I turn the text into R2L as Andreas suggested
then, yes, I see the difference. But you talked about R2L text (the
arabic words) within an L2R paragraph, not the other way around; or at
least that's how I interpret it. So obviously my lousy English skills
are failing me again.

    Juanma

Re: arrow keys vs. C-f/b/n/p

[Davis Herring]

> Yes. But I'm clearly misunderstanding something. With etc/HELLO as is,
> if I move the cursor over the arabic line, both C-f and <right> do the
> exact same thing. If I turn the text into R2L as Andreas suggested
> then, yes, I see the difference. But you talked about R2L text (the
> arabic words) within an L2R paragraph, not the other way around; or at
> least that's how I interpret it. So obviously my lousy English skills
> are failing me again.

That's just the point.  <right>, as it stands, is only aware of the
direction of the paragraph, not of the text, so it moves forward in
logical order in an L2R paragraph even when that means moving leftward on
the screen.  Eli, I believe, is proposing changing things so that
<right>/etc. move visually in all cases (while C-f/etc. would still move
logically in all cases).

Davis

-- 
This product is sold by volume, not by mass.  If it appears too dense or
too sparse, it is because mass-energy conversion has occurred during
shipping.

Re: arrow keys vs. C-f/b/n/p

[Juanma Barranquero]

On Tue, Jun 8, 2010 at 00:32, Davis Herring <herring@lanl.gov> wrote:

> That's just the point.  <right>, as it stands, is only aware of the
> direction of the paragraph, not of the text, so it moves forward in
> logical order in an L2R paragraph even when that means moving leftward on
> the screen.

Ah, OK. So what I was observing (no difference between C-f and <right>
in L2R paragraphs) is the wrong behavior. I was indeed
misunderstanding.

Thanks,

    Juanma

Re: arrow keys vs. C-f/b/n/p

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Mon, 07 Jun 2010 17:01:24 +0200
> 
> >> likely the same with "C-a" "C-e" for logical movement as contrasted
> >> to <home> and <end> for visual movement.
> >
> > To this, I object.  I made the left/right arrows and their
> > Ctrl-modified siblings move to the left resp. right, regardless of the
> > logical buffer order, because this is what users of bidirectional
> > scripts expect -- they are used to it in every other bidi-aware
> > application out there.  By contrast, <home> and <end> always move to
> > the beginning resp. end of the current line,
> 
> The left border of the screen is not necessarily the next character
> after the line feed.

Ah, that was my misunderstanding, then.  You want <home> go to the
first and <end> to the last character of the line in the visual order,
right?  Then I agree.

Re: arrow keys vs. C-f/b/n/p

[Eli Zaretskii]

> Date: Mon, 7 Jun 2010 15:32:55 -0700 (PDT)
> From: "Davis Herring" <herring@lanl.gov>
> Cc: David Kastrup <dak@gnu.org>, Andreas Schwab <schwab@linux-m68k.org>,
> 	Stefan Monnier <monnier@iro.umontreal.ca>, emacs-devel@gnu.org
> Reply-To: herring@lanl.gov
> 
> That's just the point.  <right>, as it stands, is only aware of the
> direction of the paragraph, not of the text, so it moves forward in
> logical order in an L2R paragraph even when that means moving leftward on
> the screen.

Correct.

> Eli, I believe, is proposing changing things so that
> <right>/etc. move visually in all cases (while C-f/etc. would still
> move logically in all cases).

No, I was not suggesting anything of this kind.  In fact, whatever I
want the left/right arrow keys to do in the bidirectional case is
already in the repository, and works as you describe above, i.e. it is
sensitive to the paragraph direction.

Personally, I don't see much use to a mode where the arrows move
independently of the logical (i.e. reading) order of the text.

Re: arrow keys vs. C-f/b/n/p

[Stefan Monnier]

> Ah, OK. So what I was observing (no difference between C-f and <right>
> in L2R paragraphs) is the wrong behavior. I was indeed
> misunderstanding.

No, what you're observing is the desired behavior.  All it shows is:

   Actually, the arrow keys don't do "visual motion".

which was pointed out to David in response to his statement:

   Since we opened that can of worms already by making C-f/C-b do
   logical motion as opposed to the arrow keys doing visual motion,

So, no we didn't open that can of worms.  And please let's stop the
thread here.


        Stefan

Re: arrow keys vs. C-f/b/n/p

[Miles Bader]

David Kastrup <dak@gnu.org> writes:
> Since we opened that can of worms already by making C-f/C-b do logical
> motion as opposed to the arrow keys doing visual motion

We did?  My, that's stupid...

> I should think
> it would make sense to disconnect C-n/C-p from the vertical arrow keys

God, please, no.

The arrow keys should not be special.

-Miles

-- 
Bacchus, n. A convenient deity invented by the ancients as an excuse for
getting drunk.

Re: arrow keys vs. C-f/b/n/p

[Uday S Reddy]

Stefan Monnier wrote:

> 
> So, no we didn't open that can of worms.  And please let's stop the
> thread here.

Sorry Stefan.  The can of worms is never too far away :-(

The decision is then that left, right, up and down do what their names imply, 
but Home and End do the same as C-a and C-e?

What is the "beginning-of-line"?  At the left, or the right, or "it depends"?

Likewise for "beginning-of-sentence"?

C-f moves right, and C-b moves left?  Likewise for M-f, M-b, M-d, M-<DEL>?  And 
what about C-d and <DEL> then?

C-k kills line to the right of point, or left of point?  Likewise for M-k?

Some unifying principle to arrive at all these decisions would be very welcome!

Cheers,
Uday

Re: arrow keys vs. C-f/b/n/p

[Stefan Monnier]

> The decision is then that left, right, up and down do what their names
> imply, but Home and End do the same as C-a and C-e?

David talked about decisions, whereas I don't want to.


        Stefan

Re: arrow keys vs. C-f/b/n/p

[Uday S Reddy]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>> The decision is then that left, right, up and down do what their names
>> imply, but Home and End do the same as C-a and C-e?
>
> David talked about decisions, whereas I don't want to.

Ok, fair enough.  I hope that whoever makes the decisions whenever
will be able to answer my questions.  I am in no hurry.

Cheers,
Uday

Re: Performance

[Kenichi Handa]

In article <jwvd3w3fyor.fsf-monnier+emacs@gnu.org>, Stefan Monnier <monnier@iro.umontreal.ca> writes:

> It seems that functionality-wise, the bidi code is almost as good as it
> was before bidi.  But performancewise, I experience some problems.
> I recently started to compile without -DENABLE_CHECKING and other such
> debugging code (tho still with -O0 and -g) and am seeing cases where
> cursor motion is slow.  The problem is most notable in operations such
> as C-e or C-n/C-p (tho those tend to be fast enough as long as I'm in
> column 0).
> Redisplay itself also seems to be noticeable slower at times.

Do you see the same slowness when you set
auto-composition-mode to nil?

---
handa@m17n.org

Re: arrow keys vs. C-f/b/n/p

[Chong Yidong]

Eli Zaretskii <eliz@gnu.org> writes:

> I made the left/right arrows and their Ctrl-modified siblings move to
> the left resp. right, regardless of the logical buffer order, because
> this is what users of bidirectional scripts expect -- they are used to
> it in every other bidi-aware application out there.

In other bidi-aware applications, what keys are used to move in logical
order (i.e. what C-f and C-b do now in the trunk)?

(I am still dubious about decoupling the arrow keys and C-f/C-b
keybindings.  Maybe we should provide a separate set of keybindings
instead.)

Re: arrow keys vs. C-f/b/n/p

[Stefan Monnier]

> (I am still dubious about decoupling the arrow keys and C-f/C-b
> keybindings.  Maybe we should provide a separate set of keybindings
> instead.)

It's actually not really decoupled.
It just switches between "C-f = right and C-b = left" and
"C-f = left and C-b = right" based on the paragraph's direction.
Which seems eminently meaningful since the associating between
"forward" and "right" is just based on our usual convention of
writing L2R.

Now addmitedly, the particular place where the choice between the two
forms of coupling is made is up for discussion: it could be based on the
direction of text underneath point (basically, make the arrow move
visually rather than logically), or based on the direction of the
paragraph (what we now have), or based on user preferences (default
depends on the locale).  I don't have a clear preference, but I think
that the current choice is  pretty good compromise between "no need for
customization, auto-adjusts to mixes of L2R and R2L buffers" and "still
move in logical rather than visual order".


        Stefan

Re: arrow keys vs. C-f/b/n/p

[James Cloos]

>>>>> "CY" == Chong Yidong <cyd@stupidchicken.com> writes:

CY> In other bidi-aware applications, what keys are used to move in logical
CY> order (i.e. what C-f and C-b do now in the trunk)?

I don't have a lot of stuff installed to test, but this is what I did
test:

In gedit C-f and C-b move forward and backward logically, whereas LEFT
and RIGHT move left and right visually.  I suspect that all παν語 (aka
pango) apps -- notably including most gtk/gnome apps -- will also do so.

In seamonkey, in a web-page's text input box, both the arrow keys and
the C-b and C-f keys move visually.

In gvim, both the h and l keys and the arrow keys move visually.

In Openoffice.org, the arrow keys move logically.

Someone with KDE installed should test koffice and/or kwriter.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: arrow keys vs. C-f/b/n/p

[Lennart Borgman]

On Fri, Jun 11, 2010 at 6:29 PM, James Cloos <cloos@jhcloos.com> wrote:
>
> In Openoffice.org, the arrow keys move logically.

Seems like a mistake to me. There will be lot of graphical apps where
the arrow keys move things visually. I think switching the visual
directions of the arrow will interfere badly because using them is
most certainly in low level memory (body memory) in the user.

Re: arrow keys vs. C-f/b/n/p

[James Cloos]

Oh, and my personal preference, as a l2r reader who occasionally deals
with r2l text, is for both C-f/C-b and LEFT/RIGHT to *always* move in
logical order.  The feedback that provides is useful and welcome.

The /only/ possible major modes where visual movement might be
preferable to logical would be picture-mode or modes with a
similar raison d'etre.

I can well imagine that those who deal with r2l more regularly
may prefer a different config, suggesting that perhaps it should
be configurable.

And perhaps the default should be locale-specific?

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

RE: arrow keys vs. C-f/b/n/p

[Drew Adams]

> > (I am still dubious about decoupling the arrow keys and C-f/C-b
> > keybindings.  Maybe we should provide a separate set of keybindings
> > instead.)
> 
> It's actually not really decoupled.
> It just switches between "C-f = right and C-b = left" and
> "C-f = left and C-b = right" based on the paragraph's direction.
> Which seems eminently meaningful since the associating between
> "forward" and "right" is just based on our usual convention of
> writing L2R.

I hesitate to say anything here, since I know _zero_ about R2L. But maybe it
will help somehow to expose my naivety in this regard. If not, ignore. Here
goes...

On the surface this seems wrong and overly complicated (to me). "Eminently
meaningful" it no doubt is, but it seems somehow bass ackwards. ;-) It seems
wrong for `right' to mean "left".

Why would it be wrong for `right' (right arrow) to always represent movement
toward the right, regardless of R2L/L2R?

I suppose that you want to make `C-f' move "forward", which is arguably toward
the left in R2L languages, and you want to keep the association of `C-f' with
`right' (right arrow).

But is it really important that "forward" in a command name move toward the left
in R2L? Why should "forward" necessarily mean "from text beginning toward text
end" rather than just "toward the right"? What is at stake here?

You don't change the meaning of `bobp' to `eobp' for R2L, do you? You don't swap
the meanings of "left" and "right". Why exchange "forward" and "backward" but
not "left" and right"? Just which things really need to be flipped/mirrored?

Why not leave everything as it is wrt "forward" movement, and let users
understand that "forward" always means toward the right (in both L2R and R2L)?
Wouldn't that be just as useful and a lot less confusing overall?

Why would you want `C-f' to move toward the right in one paragraph and then jump
to the "beginning" (at the right end) of the "next" paragraph and start moving
toward the left - just because it is R2L? Why wouldn't you want `C-f' (and the
`right' key and `C-M-f' and `forward-paragraph'...) to always move right,
consistently?

OK, I understand that there is a mental association of "forward" with the
direction that text is inserted, but how important is it that command names
reflect that? Will someone be confused or offended if `backward-paragraph'
always moves toward the left? Will someone complain that what they think of as
"forward" we call "backward"?

Maybe, if we kept things consistent wrt forward-means-right then we would also
want a new command/key that would jump to the "beginning" of the next paragraph,
where that "beginning" position would be a different for R2L and L2R paragraphs.
If so, then we could just add that command/key. But once you are in a paragraph,
I would think that the ordinary motion etc. commands could work normally.

IOW, why not introduce any convenient DWIM transitioning behavior only at
R2L/L2R boundaries, and leave the behavior within a piece of R2L or L2R text
alone?

Maybe you'll say that at least `C-d' and `backspace' need to flip direction
(forward/backward), so why not `C-f' also?

I have the same naive question for those keys - why not just let R2L flip the
_meaning_ of forward/backward instead of the key behavior, so that `C-d' and
`backspace' are always rightward and leftward deletion, respectively? Is it
important that the keys themselves flip?

I can see that self-insertion should move point to the left of the inserted char
in R2L. Beyond that, I don't see why all or most key behaviors need to flip.

> Now addmitedly, the particular place where the choice between the two
> forms of coupling is made is up for discussion: it could be 
> based on the direction of text underneath point (basically, make
> the arrow move visually rather than logically), or based on the
> direction of the paragraph (what we now have), or based on user
> preferences (default depends on the locale).  I don't have a clear
> preference, but I think that the current choice is  pretty good
> compromise between "no need for customization, auto-adjusts to
> mixes of L2R and R2L buffers" and "still
> move in logical rather than visual order".

Why not keep it simple and just have `C-f' and `right' always move toward the
right?  Likewise for everything that has a "forward" meaning.  Why would that be
problematic for a R2L user/language?

This sounds a bit like trying to redefine "northward" as "southward" when you
cross the equator into the southern hemisphere. What's the point?

Again, I know nothing about R2L. I'm sure it's a tough and tricky problem, and
I'm sure you have thought it all out carefully. It just seems wrong and
unnecessarily complicated for this ignoramus who has not thought it all out.

If nothing else, perhaps the naive questions expressed here will help you
document the bidi stuff for other naive users - HTH.  Perhaps something
analogous to node `(elisp) Not Intervals' would be helpful for this, explaining
the logic behind the design choices.

Re: arrow keys vs. C-f/b/n/p

[Chong Yidong]

James Cloos <cloos@jhcloos.com> writes:

> In gedit C-f and C-b move forward and backward logically, whereas LEFT
> and RIGHT move left and right visually.  I suspect that all παν語 (aka
> pango) apps -- notably including most gtk/gnome apps -- will also do so.
>
> In seamonkey, in a web-page's text input box, both the arrow keys and
> the C-b and C-f keys move visually.
>
> In gvim, both the h and l keys and the arrow keys move visually.
>
> In Openoffice.org, the arrow keys move logically.

Well, this sample indicates a complete lack of consensus, contra Eli's
earlier statement that

>>> this is what users of bidirectional scripts expect -- they are used
>>> to it in every other bidi-aware application out there.

One possibility is that editors intended specifically for bidi, such as
native-language Hebrew or Arabic apps, have a convention that vim,
gedit, etc have not taken up.  So, could anyone who has experience with
such apps speak up?

Re: arrow keys vs. C-f/b/n/p

[Lennart Borgman]

On Fri, Jun 11, 2010 at 7:37 PM, Chong Yidong <cyd@stupidchicken.com> wrote:
> James Cloos <cloos@jhcloos.com> writes:
>
>> In gedit C-f and C-b move forward and backward logically, whereas LEFT
>> and RIGHT move left and right visually.  I suspect that all παν語 (aka
>> pango) apps -- notably including most gtk/gnome apps -- will also do so.
>>
>> In seamonkey, in a web-page's text input box, both the arrow keys and
>> the C-b and C-f keys move visually.
>>
>> In gvim, both the h and l keys and the arrow keys move visually.
>>
>> In Openoffice.org, the arrow keys move logically.
>
> Well, this sample indicates a complete lack of consensus, contra Eli's
> earlier statement that
>
>>>> this is what users of bidirectional scripts expect -- they are used
>>>> to it in every other bidi-aware application out there.
>
> One possibility is that editors intended specifically for bidi, such as
> native-language Hebrew or Arabic apps, have a convention that vim,
> gedit, etc have not taken up.  So, could anyone who has experience with
> such apps speak up?


Looking around on the internet (not being a bidi reader) it seems to
me that it is mostly a programming mistake to let the arrow keys
change direction. (Somehow maybe also from a beginners perspective the
right arrow might mean "go forward".) However for someone editing
texts in some way this is awkward IMO. Here are some links:

- http://www.sfc.wide.ad.jp/~akihiko/java/jdk1.2beta3/docs/guide/2d/spec/j2d-fonts.fm1.html
(look for "moving the caret").

- http://wiki.sibawayhi.org/keyboard - READING IS NOT CURSORING!

- http://bugs.dojotoolkit.org/ticket/8037 (just a short complaints
about cursor changing their ways)

- http://www.qsm.co.il/Hebrew/logicUI22.htm - "The logical approach is
sometimes preferred by implementors, mostly because its implementation
is easier. This is not a recommended solution."

Re: arrow keys vs. C-f/b/n/p

[Uday S Reddy]

Drew Adams wrote:

>> It's actually not really decoupled.
>> It just switches between "C-f = right and C-b = left" and
>> "C-f = left and C-b = right" based on the paragraph's direction.
>> Which seems eminently meaningful since the associating between
>> "forward" and "right" is just based on our usual convention of
>> writing L2R.
> 
> On the surface this seems wrong and overly complicated (to me). "Eminently
> meaningful" it no doubt is, but it seems somehow bass ackwards. ;-) It seems
> wrong for `right' to mean "left".

No, Stefan is not saying 'right' should mean "left"!

He is saying that 'right' moves right, 'left' moves left, 'C-f' moves forward 
in the text direction and 'C-b' moves backward in the text direction. 
Sometimes the two sets of keys match up one way, and sometimes the other way. 
Eminently meaningful indeed!

Whether you want to call this "coupled" or "decoupled" is a matter of 
terminology.

> But is it really important that "forward" in a command name move toward the left
> in R2L? Why should "forward" necessarily mean "from text beginning toward text
> end" rather than just "toward the right"? What is at stake here?

Only an R2L user can answer that (which I am not).  However, consistency 
matters.  If "forward" doesn't really mean forward in the text, I think you 
will end up with nonsense in the end, such as "beginning of sentence" moving to 
the end of sentence.  That is why I came up with a bunch of questions the other 
day, which seem all interlinked.

Cheers,
Uday

Re: arrow keys vs. C-f/b/n/p

[Chong Yidong]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Now addmitedly, the particular place where the choice between the two
> forms of coupling is made is up for discussion: it could be based on
> the direction of text underneath point (basically, make the arrow move
> visually rather than logically), or based on the direction of the
> paragraph (what we now have), or based on user preferences (default
> depends on the locale).  I don't have a clear preference, but I think
> that the current choice is pretty good compromise between "no need for
> customization, auto-adjusts to mixes of L2R and R2L buffers" and
> "still move in logical rather than visual order".

My question is: do people who edit bidi frequently need or expect a
separate command for moving logically?

If not, we could require C-f/[right] to always move right, and
C-b/[left] to always move left.  This would be the right-char and
left-char commands, with `forward-char' and `backward-char' no longer
bound to keys.

Re: arrow keys vs. C-f/b/n/p

[Chong Yidong]

Chong Yidong <cyd@stupidchicken.com> writes:

> My question is: do people who edit bidi frequently need or expect a
> separate command for moving logically?

To be precise, I meant a separate command for moving *by one character*
logically.

Also, I would like to know what existing bidi editors do for
Ctrl-left/right.

RE: arrow keys vs. C-f/b/n/p

[Drew Adams]

> >> It's actually not really decoupled.
> >> It just switches between "C-f = right and C-b = left" and
> >> "C-f = left and C-b = right" based on the paragraph's direction.
> >> Which seems eminently meaningful since the associating between
> >> "forward" and "right" is just based on our usual convention of
> >> writing L2R.
> > 
> > On the surface this seems wrong and overly complicated (to 
> > me). "Eminently meaningful" it no doubt is, but it seems
> > somehow bass ackwards. ;-) It seems wrong for `right' to
> > mean "left".
> 
> No, Stefan is not saying 'right' should mean "left"!

When I used `...' quotes for `right', I was referring to the `right' key, aka
<right> aka "right arrow".  The rest of my post also makes this clear, I think.

> He is saying that 'right' moves right, 'left' moves left, 
> 'C-f' moves forward in the text direction and 'C-b' moves
> backward in the text direction. 

And the rest of my post makes clear that I understood that.

> Sometimes the two sets of keys match up one way, and 
> sometimes the other way. Eminently meaningful indeed!
> Whether you want to call this "coupled" or "decoupled"
> is a matter of terminology.

I did not refer to either "coupled" or "decoupled".
I don't care what one calls it.
 
> > But is it really important that "forward" in a command name 
> > move toward the left in R2L?  Why should "forward"
> > necessarily mean "from text beginning toward text
> > end" rather than just "toward the right"? What is at stake here?
> 
> Only an R2L user can answer that (which I am not).

Well, that's the question.  Why flip the notions of "forward" and "backward" but
not bob and eob?  Why flip `C-f' but not `right' (or vice versa)?

Why should a change in text-insertion direction cause lots of directional
keys/commands to flip (but not all, apparently)?

> However, consistency matters.

Consistency matters.  But which consistency?

> If "forward" doesn't really mean forward in the text,

What about forward in the _buffer_?

Should `forward-char' reflect the R2L/L2R quality of the surrounding chars, or
should it just reflect the buffer directionality (bob -> eob)?

What about beginning of buffer? Should it sometimes become end of buffer? Should
that happen only if the entire buffer is R2L?

Anyone smell a can of worms around here?

> I think you will end up with nonsense in the end, such as "beginning of 
> sentence" moving to the end of sentence.

A sentence, like a defun, has a well-defined start and end. Any display or
editing mode should easily DTRT wrt such things. You could even have a mode
where a sentence is represented as a tree. End-of-sentence would still be
unambiguous, wherever and however it was displayed.

My question is about the "forward" direction. Should it be the traditional,
buffer-oriented direction or should it change depending on the current paragraph
(surrounding text) etc.?

Naively, it seems simpler to have movement from bob toward eob be "forward",
with (1+ (point-min)) being farther "forward" than (point-min), ... and
(point-max) being the farthest "forward".

Of course, anyone and any mode is free to define a custom "forward" direction or
movement.  thingatpt.el offers one way to do that for different kinds of things,
for example. So I'm not arguing that one should not be able to have different
"forward" behaviors or notions.

But for the ordinary cursor movements, I still raise the naive question: Why not
just let "forward" be toward eob?

> That is why I came up with a bunch of questions the other 
> day, which seem all interlinked.

Re: arrow keys vs. C-f/b/n/p

[W Dan Meyer]

Lennart Borgman <lennart.borgman@gmail.com> writes:
>>
>> One possibility is that editors intended specifically for bidi, such as
>> native-language Hebrew or Arabic apps, have a convention that vim,
>> gedit, etc have not taken up.  So, could anyone who has experience with
>> such apps speak up?
>
>
> Looking around on the internet (not being a bidi reader) it seems to
> me that it is mostly a programming mistake to let the arrow keys
> change direction. (Somehow maybe also from a beginners perspective the
> right arrow might mean "go forward".) However for someone editing
> texts in some way this is awkward IMO. Here are some links:
>
> - http://www.sfc.wide.ad.jp/~akihiko/java/jdk1.2beta3/docs/guide/2d/spec/j2d-fonts.fm1.html
> (look for "moving the caret").
>
> - http://wiki.sibawayhi.org/keyboard - READING IS NOT CURSORING!
>
> - http://bugs.dojotoolkit.org/ticket/8037 (just a short complaints
> about cursor changing their ways)
>
> - http://www.qsm.co.il/Hebrew/logicUI22.htm - "The logical approach is
> sometimes preferred by implementors, mostly because its implementation
> is easier. This is not a recommended solution."

Nice reesearch.

I agree with those findings.

For me it looks all right to have those forward-backward bindings. For
few reasons:

- C-{f,b} is used mostly by experienced Emacs users, if they don't like
  they will most likely change it (I am sure you can contradict it, I
  have never learnt to use them)

- Obviously it was said before, forward-backward means forward-backward
  not left-right. 

- We have a choice, we can use C-{f,b} to preserve the logical
  direction, plus we add the functonality to rebind it if is
  needed. Question if that should be mode, global state and a set of
  functions

- if it will not work as expected (I think there must be an agreement
  how the programs behave in the right-left direction over the world),
  we will get kind e-mails saying so (I hope kind ;) )

- everything dependes on a person, if somebody is not a native
  speaker/writer in such language and writes a letter probably it will be
  cumbersome to use reversed bindings, it is easier to read the text
  read-left than swapping keys (muscle memory, and the fact one is forced
  to read, it cannot be done in a different way).
 
So from my point of view maybe good idea is to:

- Let the user swap arrows too (or the underlaying implementation, well it
  can be done by binding just different symbol),

- Find a person which uses bidi, not in Emacs but generally and ask if
  he/she would prefered this behaviour. 

- Switching even OpenOffice to the right-left mode and check the default

Just mine 2 cents.

Wojciech

RE: arrow keys vs. C-f/b/n/p

[Uday S Reddy]

Drew Adams writes:

> When I used `...' quotes for `right', I was referring to the `right'
> key, aka <right> aka "right arrow".  The rest of my post also makes
> this clear, I think.

Sorry I misunderstood.  Actually, I find most of this discussion hard
to understand (as also the line-move-visual discussion earlier).  It
is best to be as clear as possible.

> > Only an R2L user can answer that (which I am not).
> 
> Well, that's the question.  Why flip the notions of "forward" and
> "backward" but not bob and eob?  Why flip `C-f' but not `right' (or
> vice versa)?
> 
> Why should a change in text-insertion direction cause lots of directional
> keys/commands to flip (but not all, apparently)?
> 
> > However, consistency matters.
> 
> Consistency matters.  But which consistency?

The consistency I am asking for is that "forward" should mean the same
thing all the time and "end" should mean the same thing all the time.
I also think "forward" and "end" should be correlated because nobody
wants to go forward to the beginning!

On the other hand, there is no need for "forward" and "right" to be
correlated.  They are independent concepts.

---

James Cloos wrote an informative message later, at 5:12pm, where he
pointed out that an L2R user that is occasionally using R2L text may
have a very different preference from some one that regularly deals
with R2L text.  So, flexibility is called for.  If we go with a
preconceived idea that C-f should always mean the same as <right>,
then we are likely to go wrong.

I might have said this earlier, but key bindings are not a big deal.
The users can always bind the keys to their liking.  But the
*functionality* should be there for them to bind the keys to.  If
there is only one function called forward-char then C-f and <right>
will get coupled, and the flexibility won't be there for the users.

If there is a forward-char and a right-char, then James Cloos can bind
both C-f and <right> to forward-char.  He gets the logical motion that
he prefers.  Some other user might want <right> to mean right.  He
might bind <right> to right-char.

> A sentence, like a defun, has a well-defined start and end. Any display or
> editing mode should easily DTRT wrt such things. You could even have a mode
> where a sentence is represented as a tree. End-of-sentence would still be
> unambiguous, wherever and however it was displayed.

Ok, that is nice.  But then, will `forward-sentence' move forward over
a sentence, while `forward-char' will move back?

That might in fact be what some users want.  Somebody might have his
neurons already wired to correlate C-f with right and C-b with left.
But we can't say that every user should follow that choice?

> My question is about the "forward" direction. Should it be the
> traditional, buffer-oriented direction or should it change depending
> on the current paragraph (surrounding text) etc.?
> 
> Naively, it seems simpler to have movement from bob toward eob be "forward",
> with (1+ (point-min)) being farther "forward" than (point-min), ... and
> (point-max) being the farthest "forward".

My belief is that many users will want "forward" to be determined by
the kind of text they are editing.  

bob and eob are harder issues.  So, I don't think we should let them
guide everything else.  Rather, they should be postponed to the end,
after everything else is decided.

In fact, it is not clear to me that when a user does bob or eob, they
necessarily want to go to the very beginning or very end of the text.
Somewhere close by is good enough.  So, if bob and eob don't work
perfectly, it doesn't matter all that much.

Cheers,
Uday

RE: arrow keys vs. C-f/b/n/p

[Drew Adams]

> > > However, consistency matters.
> > 
> > Consistency matters.  But which consistency?
> 
> The consistency I am asking for is that "forward" should mean the same
> thing all the time and "end" should mean the same thing all the time.

Forward and end wrt what?

What I'm suggesting is that we might want to keep the "what" to be the buffer,
in general. And forward in the buffer means farther from the beginning
(point-min) and closer to the end (point-max).

That's really the question. What are "forward" and "end" relative to, and for
which commands? Is a command such as `forward-char' about moving forward in the
buffer or about moving forward within the immediately surrounding text (which
might be R2L)?

> I also think "forward" and "end" should be correlated because nobody
> wants to go forward to the beginning!

Same question. If the current sentence is R2L then moving forward in the buffer,
i.e. toward the end (of the buffer), also means moving backward in the sentence,
i.e. toward the beginning (of the sentence).

> On the other hand, there is no need for "forward" and "right" to be
> correlated.  They are independent concepts.

Well, that's why I rephrased "right" as forward-in-the-buffer, to make clearer
what is the real question (that I raised).

> James Cloos wrote an informative message later, at 5:12pm, where he
> pointed out that an L2R user that is occasionally using R2L text may
> have a very different preference from some one that regularly deals
> with R2L text.  So, flexibility is called for.  If we go with a
> preconceived idea that C-f should always mean the same as <right>,
> then we are likely to go wrong.

And if we go with a preconceived idea that C-f should change directions
depending on the text surrounding the cursor (R2L vs L2R), then we are also
likely to go wrong.  Consistency, OK, but which?

Assuming we don't change anything that affects someone who never uses R2L, I
won't complain for myself, as it won't really affect me.

But I raised the question because I get the impression we might be marching off
into the swamp.  And also because I would like the doc to explain these things
clearly, in the end.

> I might have said this earlier, but key bindings are not a big deal.
> The users can always bind the keys to their liking.  But the
> *functionality* should be there for them to bind the keys to.  If
> there is only one function called forward-char then C-f and <right>
> will get coupled, and the flexibility won't be there for the users.
> 
> If there is a forward-char and a right-char, then James Cloos can bind
> both C-f and <right> to forward-char.  He gets the logical motion that
> he prefers.  Some other user might want <right> to mean right.  He
> might bind <right> to right-char.

Certainly we will need various commands and offer flexibility.

I think this is about (a) the default bindings and (b) the behavior in general
of `forward-'* commands. 

IIUC, the "coupling" question is part of (a): Should the default bindings be
different for `right' (`<right>') and `C-f'? Should one of them DTRT for R2L but
the other not?

> Ok, that is nice.  But then, will `forward-sentence' move forward over
> a sentence, while `forward-char' will move back?

It might, if the sentence text is written "backwards" wrt the buffer direction.

Again, that's the question. Does `forward-char' respect the directionality of
the buffer or the directionality of the text surrounding point?

BTW, what do we do when point is at a boundary (R2L/L2R or L2R/R2L)?  Which way
is forward?  Should that depend on the previous cursor movement (where we're
coming from)? Surely, that way lies madness. ;-)

I imagine that these questions are not new for R2L folks. Hasn't some sort of
standard behavior been worked out for buffers/documents that combine R2L and L2R
text? If so, how does that conventional behavior fit with Emacs?

> That might in fact be what some users want.  Somebody might have his
> neurons already wired to correlate C-f with right and C-b with left.
> But we can't say that every user should follow that choice?

We can't say anything about every user.
We do have to decide something about the default behavior, however.

> > My question is about the "forward" direction. Should it be the
> > traditional, buffer-oriented direction or should it change depending
> > on the current paragraph (surrounding text) etc.?
> > 
> > Naively, it seems simpler to have movement from bob toward 
> > eob be "forward", with (1+ (point-min)) being farther "forward"
> > than (point-min), ... and (point-max) being the farthest "forward".
> 
> My belief is that many users will want "forward" to be determined by
> the kind of text they are editing.

You're speaking only about key bindings, I think. I'm asking also about commands
and command names.  Should most `forward-'* commands reverse direction when the
text around point is R2L?

Will `forward-char' be DWIMed to fit R2L (as opposed to some other command being
defined for that)?  How would that affect Lisp code and keyboard macros?

> bob and eob are harder issues.  So, I don't think we should let them
> guide everything else.  Rather, they should be postponed to the end,
> after everything else is decided.
> 
> In fact, it is not clear to me that when a user does bob or eob, they
> necessarily want to go to the very beginning or very end of the text.
> Somewhere close by is good enough.  So, if bob and eob don't work
> perfectly, it doesn't matter all that much.

I don't imagine that anyone is suggesting to change `beginning-of-buffer' so it
goes to eob.  My question is about the behavior and notion of "forward" in
general.

At any rate, I've admitted that I know _nothing_ about R2L, and you've said that
you too are not an R2L user. So what we two have to say about this is perhaps
not so important, and I'll stop here. Eventually someone who knows better will
chime in and set things straight. ;-)

Re: arrow keys vs. C-f/b/n/p

[Miles Bader]

W Dan Meyer <wojciech.meyer@googlemail.com> writes:
> - C-{f,b} is used mostly by experienced Emacs users, if they don't like
>   they will most likely change it (I am sure you can contradict it, I
>   have never learnt to use them)

Even if this is true (I dunno), I don't think we want make it even
_harder_ for new users to learn to use Emacs' better bindings......

-Miles

-- 
Love is a snowmobile racing across the tundra.  Suddenly it flips over,
pinning you underneath.  At night the ice weasels come.  --Nietzsche

Re: arrow keys vs. C-f/b/n/p

[Miles Bader]

James Cloos <cloos@jhcloos.com> writes:
> Oh, and my personal preference, as a l2r reader who occasionally deals
> with r2l text, is for both C-f/C-b and LEFT/RIGHT to *always* move in
> logical order.  The feedback that provides is useful and welcome.

I agree.

All of this seems like excessive hand-wringing and premature
optimization to me.  In general I think it's good to keep things simple
and consistent to begin with, and only start adding complications if
there's solid user demand...

-Miles

-- 
"Nah, there's no bigger atheist than me.  Well, I take that back.
I'm a cancer screening away from going agnostic and a biopsy away
from full-fledged Christian."  [Adam Carolla]

Re: arrow keys vs. C-f/b/n/p

[Eli Zaretskii]

> From: Miles Bader <miles@gnu.org>
> Date: Sat, 12 Jun 2010 14:43:24 +0900
> Cc: Chong Yidong <cyd@stupidchicken.com>, David Kastrup <dak@gnu.org>,
> 	Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> 
> James Cloos <cloos@jhcloos.com> writes:
> > Oh, and my personal preference, as a l2r reader who occasionally deals
> > with r2l text, is for both C-f/C-b and LEFT/RIGHT to *always* move in
> > logical order.  The feedback that provides is useful and welcome.
> 
> I agree.

This thread is riddled with confusion and misunderstanding, not the
least part of which is because many participants do not read any R2L
scripts.

I will respond to individual messages, but first let me set the stage
by defining the terminology and the use-cases.  I'm sorry, but this
message will necessarily be a longish one.  Nevertheless, please read
it in its entirety before responding to any further messages.

The current Emacs supports two text directions and two paragraph
directions.  Both can be either left-to-right (L2R) or right-to-left
(R2L).

Paragraph direction determines whether the text starts at the left
window margin or at the right margin.  Text direction determines
the direction in which text is read.

A L2R paragraph can include both L2R text and R2L text.  Likewise, a
R2L paragraph can include both L2R text and R2L text.

In general, a R2L paragraph is supposed to have most of its text R2L,
and a L2R paragraph is supposed to have most of its text L2R, but
that's not a restriction, and Emacs does not impose that in any way.
It's just that a L2R paragraph that has most or all of its text R2L
will look awkward; try setting bidi-paragraph-direction to
right-to-left in the "*scratch*" buffer, and you'll see what I mean.

Here are 2 use-cases I will use throughout the rest of the discussion,
and I suggest that everyone else does as well, to avoid confusion.  As
usual, upper-case letters represent characters from R2L scripts (such
as Arabic or Hebrew), while lower-case letters represent characters
from L2R scripts, such as the various Latin scripts.

Here's how a L2R paragraph which includes mixed L2R and R2L text will
be shown on the screen:

  abcde EDCBA xyz

This display corresponds to the following buffer text:

  abcde ABCDE xyz

Here's a R2L paragraph that includes mixed R2L and L2R text.  Note
that such a paragraph is displayed flushed all the way to the right
margin of the window:

                                       ZYX abcde EDCBA

This display corresponds to the following buffer text:

  ABCDE abcde XYZ

The buffer order of characters is known as the "logical" or "reading"
order.  The order of characters on the screen is known as the "visual"
order.  Note that the visual order always goes from left to right, at
least in Emacs (and hence in this thread).  That is, in the R2L
paragraph example above, the first character in the visual order is Z,
the second is Y, etc.

This thread is about movement keys.  In Emacs, these keys always move
in the _logical_ order, i.e. in the order of the character positions
in the buffer.  While visual-mode movement might make sense in some
use-cases, it is not yet implemented.  Personally, I think it is much
less important (see my other messages for why I think so), which is
the main reason why they are not yet implemented in Emacs.  But I hope
everyone will agree that having logical-order motion keys is an
absolute must.

So let's stop talking about visual-order movement vs logical-order
movement, from this point on, in this thread.  The _only_ movement
mode we have in Emacs is the logical-order one.  Both C-f/C-b and the
arrow keys move strictly in logical order.

C-f always moves forward in the logical order.  Thus, in the first
example above, it will move like this:

  a->b->c->d->e->A->B->C->D->E->x->y->z

Note that this means the cursor will jump on the screen between e and
A and then jump again from E to x.

In the second example, of the R2L paragraph, C-f will move cursor like
this:

                               Z<-Y<-X<-e<-d<-c<-b<-a<-E<-D<-C<-B<-A

Note that again, it will jump from E to a and from e to X.

Now please note an important detail: in a L2R paragraph, C-f generally
moves cursor _to_the_right_, even though it could sometimes change
direction and move to the left, when we are traversing R2L text
embedded into a L2R paragraph.  Similarly, in a R2L paragraph, C-f
generally moves cursor _to_the_left_.  In the important special case,
when L2R paragraphs include only L2R text and R2L paragraphs include
only R2L text, cursor motion with C-f is strictly to the right
resp. to the left.

This is the reason why the <right> arrow key moves like C-f in a L2R
paragraph, and why the <left> arrow key moves like C-f in a R2L
paragraph.  (And similarly with C-b.)  It is so that Emacs behaves as
expected in a purely R2L paragraph containing only R2L text.  Any
other operation would be confusingly counter-intuitive: the <right>
key would actually move cursor _to_the_left_!

Again, we are talking _only_ about _logical-order_movement_!

I hope the issue is more clear now.  I will now go and respond to some
of the comments posted in this thread.

Re: arrow keys vs. C-f/b/n/p

[Eli Zaretskii]

> X-Spam-Status: No, score=0.1 required=5.0 tests=BAYES_00,GAPPY_SUBJECT
> 	autolearn=no version=3.3.1
> From: Chong Yidong <cyd@stupidchicken.com>
> Cc: David Kastrup <dak@gnu.org>, emacs-devel@gnu.org
> Date: Fri, 11 Jun 2010 10:12:39 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I made the left/right arrows and their Ctrl-modified siblings move to
> > the left resp. right, regardless of the logical buffer order, because
> > this is what users of bidirectional scripts expect -- they are used to
> > it in every other bidi-aware application out there.
> 
> In other bidi-aware applications, what keys are used to move in logical
> order (i.e. what C-f and C-b do now in the trunk)?

Most of those I know about don't have movement keys in addition to
arrows.

> I am still dubious about decoupling the arrow keys and C-f/C-b
> keybindings.

Can you say why?

Re: arrow keys vs. C-f/b/n/p

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>,  David Kastrup <dak@gnu.org>,  emacs-devel@gnu.org
> Date: Fri, 11 Jun 2010 11:47:41 -0400
> 
> > (I am still dubious about decoupling the arrow keys and C-f/C-b
> > keybindings.  Maybe we should provide a separate set of keybindings
> > instead.)
> 
> It's actually not really decoupled.
> It just switches between "C-f = right and C-b = left" and
> "C-f = left and C-b = right" based on the paragraph's direction.
> Which seems eminently meaningful since the associating between
> "forward" and "right" is just based on our usual convention of
> writing L2R.

Exactly.

> Now addmitedly, the particular place where the choice between the two
> forms of coupling is made is up for discussion: it could be based on the
> direction of text underneath point (basically, make the arrow move
> visually rather than logically)

There's already infrastructure in the display engine to support such a
feature.  Every glyph we produce for display is marked with its
bidirectional level.  We can base all kinds of feature on that in the
future if we decide.

> I don't have a clear preference, but I think that the current choice
> is pretty good compromise between "no need for customization,
> auto-adjusts to mixes of L2R and R2L buffers" and "still move in
> logical rather than visual order".

That's also my hope.  And until and unless we hear otherwise from
users of bidirectional scripts, I don't recommend considering any
changes in that.

Re: arrow keys vs. C-f/b/n/p

[Eli Zaretskii]

> From: James Cloos <cloos@jhcloos.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  David Kastrup <dak@gnu.org>,  emacs-devel@gnu.org
> Copyright: Copyright 2009 James Cloos
> OpenPGP-Fingerprint: E9E9 F828 61A4 6EA9 0F2B  63E7 997A 9F17 ED7D AEA6
> Date: Fri, 11 Jun 2010 12:29:31 -0400
> 
> In gedit C-f and C-b move forward and backward logically, whereas LEFT
> and RIGHT move left and right visually.

Please be more specific, after reading my message where I presented
the definitions relevant to this thread.  Emacs currently implements
only the logical-mode motion, both with C-f/C-b and with arrow keys.

> In seamonkey, in a web-page's text input box, both the arrow keys and
> the C-b and C-f keys move visually.

A useless mode, if you ask me.  It might be good enough for a Web
browser, but it's useless in a text editor.  That is because, with
visual-mode only, you have no good way of marking the region (which
must always be marked in logical buffer order).  Marking region and
features based on the region are so basic in Emacs that without
logical-order motion keys Emacs will be useless.

> In gvim, both the h and l keys and the arrow keys move visually.

Please don't consider Vim a bidi-aware application.  All Vim does is
simple reversal of text on the screen.  It has no idea what to reverse
and what not, and neither does it support UAX#9.  It's small wonder it
can only support visual-order movement.

> In Openoffice.org, the arrow keys move logically.

Again, please be more specific: what does the left arrow key do in a
R2L paragraph?  Does it go backwards in the text, eventually ending up
at the first text character, or does it go forward?

Re: arrow keys vs. C-f/b/n/p

[Eli Zaretskii]

> From: Lennart Borgman <lennart.borgman@gmail.com>
> Date: Fri, 11 Jun 2010 19:09:44 +0200
> Cc: Chong Yidong <cyd@stupidchicken.com>, Eli Zaretskii <eliz@gnu.org>, David Kastrup <dak@gnu.org>, 
> 	emacs-devel@gnu.org
> 
> On Fri, Jun 11, 2010 at 6:29 PM, James Cloos <cloos@jhcloos.com> wrote:
> >
> > In Openoffice.org, the arrow keys move logically.
> 
> Seems like a mistake to me.

What is the mistake?  What do you think James described by the above
sentence?

> I think switching the visual directions of the arrow will interfere
> badly because using them is most certainly in low level memory (body
> memory) in the user.

What program do you think switched the visual directions of the arrow
keys?  Emacs doesn't do that: the <right> key moves to the right, and
the <left> key moves to the left.  (They didn't, before I made the
changes in the bindings of the arrow keys.)

Re: arrow keys vs. C-f/b/n/p

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Cc: Eli Zaretskii <eliz@gnu.org>,  David Kastrup <dak@gnu.org>,  emacs-devel@gnu.org
>> Date: Fri, 11 Jun 2010 11:47:41 -0400
>> 
>> > (I am still dubious about decoupling the arrow keys and C-f/C-b
>> > keybindings.  Maybe we should provide a separate set of keybindings
>> > instead.)
>> 
>> It's actually not really decoupled.
>> It just switches between "C-f = right and C-b = left" and
>> "C-f = left and C-b = right" based on the paragraph's direction.
>> Which seems eminently meaningful since the associating between
>> "forward" and "right" is just based on our usual convention of
>> writing L2R.
>
> Exactly.
>
> That's also my hope.  And until and unless we hear otherwise from
> users of bidirectional scripts, I don't recommend considering any
> changes in that.

I use left and right cursor keys (actually, all cursor keys) together
with autorepeat for visual navigation: you press the key down, follow
the movement, and stop when you are pretty much there.  Logical movement
is not good for that.

-- 
David Kastrup

Re: arrow keys vs. C-f/b/n/p

[Eli Zaretskii]

> From: "Drew Adams" <drew.adams@oracle.com>
> Cc: "'Eli Zaretskii'" <eliz@gnu.org>, "'David Kastrup'" <dak@gnu.org>,
>         <emacs-devel@gnu.org>
> Date: Fri, 11 Jun 2010 10:12:45 -0700
> 
> On the surface this seems wrong and overly complicated (to me). "Eminently
> meaningful" it no doubt is, but it seems somehow bass ackwards. ;-)

Do you still think that, after reading my explanations?

> It seems wrong for `right' to mean "left".

It does seem wrong.  Which is why Emacs doesn't do that.  Do you
agree, after reading my explanations?

> Why would it be wrong for `right' (right arrow) to always represent movement
> toward the right, regardless of R2L/L2R?

It is not wrong, which is why Emacs behaves precisely like that.

The only exception is when R2L text is embedded in a L2R paragraph
(i.e. in otherwise mostly L2R text).  Since arrow keys still move in
logical order, the direction is reversed while we are traversing R2L
text; it returns to its usual sense when point is back inside L2R
text.

Having the right arrow always go right (i.e. in the visual order)
means that users will be unable to press the Shift key and have the
text they traverse be marked correctly.  That is why we have only the
logical-order movement for now -- because it is indispensable.  While
I don't object to have a visual-order movement mode, I think it is
much less useful, and it must be a separate mode in any case.

In general, I believe the visual-mode movement is useful only to
people who don't really read the bidirectional text they traverse with
the cursor-motion keys.  Any person who actually reads and understand
the text will need the logical-order motion first and foremost, and
the fact that direction is sometimes switched for a short period of
time is expected and understood by users of bidirectional scripts.  It
is a necessary complication, and one that is not hard to become used
to -- because the cursor moves in the reading order, the same order
our eyes move when we scan text.

> I suppose that you want to make `C-f' move "forward", which is arguably toward
> the left in R2L languages, and you want to keep the association of `C-f' with
> `right' (right arrow).

Why would I want C-f to be associated with <right> if it actually
moves to the left?  Don't you think it's terribly confusing mentally?

> But is it really important that "forward" in a command name move toward the left
> in R2L? Why should "forward" necessarily mean "from text beginning toward text
> end" rather than just "toward the right"? What is at stake here?

At stake here is a very fundamental issue: the reading order is still
from character position 1 to position 2 to 3 and so forth.  The order
on the screen does not change this fundamental fact.  If you break
this fundamental relation between the buffer order and the reading
order, users will throw Emacs out the window.

> You don't change the meaning of `bobp' to `eobp' for R2L, do you? You don't swap
> the meanings of "left" and "right". Why exchange "forward" and "backward" but
> not "left" and right"?

We didn't change forward and backward.  We did change the relation
between left/right and forward/backward.  That is, "right" is not
always "forward" anymore.

> Just which things really need to be flipped/mirrored?

We need to break the one-to-one relation between buffer's logical
order and the left/right direction.  This is what bidirectional text
support is all about: "leftmost" no longer means "the first", because
screen direction and screen sides no longer correspond to buffer
order, in general.

> Why not leave everything as it is wrt "forward" movement, and let users
> understand that "forward" always means toward the right (in both L2R and R2L)?
> Wouldn't that be just as useful and a lot less confusing overall?

No, it would be terribly confusing.  "Forward" must mean what the word
means, and "left" must also mean what the word says.  Any other way
means placing a terrible mental burden on the user -- and for what?
for convenience of ELisp programmers?

> Maybe you'll say that at least `C-d' and `backspace' need to flip direction
> (forward/backward)

They don't, and shouldn't, IMO.  C-d always deletes the character at
point, and Backspace always deletes the character before point.  The
basic Emacs notions of "before point" and "after point" are not
changed by the bidi support, because doing that would mean that
everything in Emacs would stop working.

In general, the basic design of the bidi support changes only the
display features.  It changes nothing in how buffer text is processed
and accessed.  This is done on purpose: I strongly believe that
bidirectional text is a display-only feature.  It must not change
anything else.

> I can see that self-insertion should move point to the left of the inserted char
> in R2L.

This is a misunderstanding.  There's no "left" and "right" in the
buffer.  Insertion doesn't move point to the left, it moves point
farther from the beginning of the buffer, i.e. it increases its buffer
position.  Whether this is right or left on the screen depends on the
surrounding bidirectional context.  I could even concoct an example
where inserting a single character reorders many characters around the
insertion locus and moves the cursor far away from its previous
position.

> Again, I know nothing about R2L. I'm sure it's a tough and tricky problem, and
> I'm sure you have thought it all out carefully. It just seems wrong and
> unnecessarily complicated for this ignoramus who has not thought it all out.

Complicated, yes.  But also necessary.  Any other solution is simply
wrong, from the user perspective.

> If nothing else, perhaps the naive questions expressed here will help you
> document the bidi stuff for other naive users - HTH.

Naive users who read bidirectional scripts do not need this to be
explained: they expect this behavior by default.

> Perhaps something analogous to node `(elisp) Not Intervals' would be
> helpful for this, explaining the logic behind the design choices.

We don't explain design choices in the manuals.  There're plenty of
comments in the code to explain these decisions; if you or someone
else can point out where more commentary is in order, I'd be more than
happy to add whatever is missing, being a documentation maniac.

Re: arrow keys vs. C-f/b/n/p

[Eli Zaretskii]

> From: Chong Yidong <cyd@stupidchicken.com>
> Date: Fri, 11 Jun 2010 13:37:59 -0400
> Cc: Eli Zaretskii <eliz@gnu.org>, David Kastrup <dak@gnu.org>,
> 	emacs-devel@gnu.org
> 
> James Cloos <cloos@jhcloos.com> writes:
> 
> > In gedit C-f and C-b move forward and backward logically, whereas LEFT
> > and RIGHT move left and right visually.  I suspect that all παν語 (aka
> > pango) apps -- notably including most gtk/gnome apps -- will also do so.
> >
> > In seamonkey, in a web-page's text input box, both the arrow keys and
> > the C-b and C-f keys move visually.
> >
> > In gvim, both the h and l keys and the arrow keys move visually.
> >
> > In Openoffice.org, the arrow keys move logically.
> 
> Well, this sample indicates a complete lack of consensus, contra Eli's
> earlier statement that

I don't think so.  At least, we should remove gvim from this sample,
and wait for James to provide more accurate information about the
rest.

> >>> this is what users of bidirectional scripts expect -- they are used
> >>> to it in every other bidi-aware application out there.
> 
> One possibility is that editors intended specifically for bidi, such as
> native-language Hebrew or Arabic apps, have a convention that vim,
> gedit, etc have not taken up.  So, could anyone who has experience with
> such apps speak up?

I just did speak up.  What is implemented in Emacs now is precisely
what those native apps do.  Why don't you believe me?

Re: arrow keys vs. C-f/b/n/p

[Eli Zaretskii]

> From: James Cloos <cloos@jhcloos.com>
> Date: Fri, 11 Jun 2010 13:12:37 -0400
> Cc: Eli Zaretskii <eliz@gnu.org>, David Kastrup <dak@gnu.org>,
> 	emacs-devel@gnu.org
> 
> Oh, and my personal preference, as a l2r reader who occasionally deals
> with r2l text, is for both C-f/C-b and LEFT/RIGHT to *always* move in
> logical order.

That's what Emacs does now.  Both C-f/C-b and the arrows move in the
logical order.  There's no visual-order movement in Emacs (yet).

Re: arrow keys vs. C-f/b/n/p

[Eli Zaretskii]

> From: Uday S Reddy <u.s.reddy@cs.bham.ac.uk>
> Date: Fri, 11 Jun 2010 20:34:46 +0100
> 
> No, Stefan is not saying 'right' should mean "left"!
> 
> He is saying that 'right' moves right, 'left' moves left, 'C-f' moves forward 
> in the text direction and 'C-b' moves backward in the text direction. 
> Sometimes the two sets of keys match up one way, and sometimes the other way. 

Exactly.

> > But is it really important that "forward" in a command name move toward the left
> > in R2L? Why should "forward" necessarily mean "from text beginning toward text
> > end" rather than just "toward the right"? What is at stake here?
> 
> Only an R2L user can answer that (which I am not).

And I am.  And I did.

> However, consistency matters.  If "forward" doesn't really mean
> forward in the text, I think you will end up with nonsense in the
> end, such as "beginning of sentence" moving to the end of sentence.

100% agreement.

Re: arrow keys vs. C-f/b/n/p

[Eli Zaretskii]

> From: Chong Yidong <cyd@stupidchicken.com>
> Cc: Eli Zaretskii <eliz@gnu.org>, David Kastrup <dak@gnu.org>,
>         emacs-devel@gnu.org
> Date: Fri, 11 Jun 2010 15:45:22 -0400
> 
> My question is: do people who edit bidi frequently need or expect a
> separate command for moving logically?

Please explain what you mean here.  _All_ movement commands in Emacs
move logically, both C-f/C-b and the arrow keys.

> If not, we could require C-f/[right] to always move right, and
> C-b/[left] to always move left.  This would be the right-char and
> left-char commands, with `forward-char' and `backward-char' no longer
> bound to keys.

You are in effect saying that screen direction is more important than
the reading order.  This is IMO fundamentally wrong.

Re: arrow keys vs. C-f/b/n/p

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

> In general, I believe the visual-mode movement is useful only to
> people who don't really read the bidirectional text they traverse with
> the cursor-motion keys.

I almost never use a single left/right cursor-motion keypress.  And when
I traverse material with the cursor-motion keys, I don't read it _while_
traversing.  I get glimpses, like when using autorepeat with
scroll-up/scroll-down.  My autorepeat rate is about 20 characters per
second.  At that speed, I can't really follow forward/backward jumps.

So no, I "don't really read the bidirectional text I traverse with the
cursor-motion keys", and I want to see a single person either claim that
she presses the cursor key once for every character she reads, or that
she reads text exactly at autorepeat speed, including jumping backwards
and forwards in mixed L2R/R2L text.

My usage patterns for C-f/C-b use less autorepeat.  Don't ask me why.

-- 
David Kastrup

Re: arrow keys vs. C-f/b/n/p

[Eli Zaretskii]

> From: Chong Yidong <cyd@stupidchicken.com>
> Cc: Eli Zaretskii <eliz@gnu.org>, David Kastrup <dak@gnu.org>,
>         emacs-devel@gnu.org
> Date: Fri, 11 Jun 2010 15:50:39 -0400
> 
> Chong Yidong <cyd@stupidchicken.com> writes:
> 
> > My question is: do people who edit bidi frequently need or expect a
> > separate command for moving logically?
> 
> To be precise, I meant a separate command for moving *by one character*
> logically.

Are you suggesting to have forward-char go backwards in the buffer
when in R2L text?

> Also, I would like to know what existing bidi editors do for
> Ctrl-left/right.

Same as what Emacs does now.

Re: arrow keys vs. C-f/b/n/p

[James Cloos]

>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:

>> In gedit C-f and C-b move forward and backward logically, whereas LEFT
>> and RIGHT move left and right visually.

EZ> Please be more specific, after reading my message where I presented
EZ> the definitions relevant to this thread.  Emacs currently implements
EZ> only the logical-mode motion, both with C-f/C-b and with arrow keys.

I used logically and visually correctly.

>> In seamonkey, in a web-page's text input box, both the arrow keys and
>> the C-b and C-f keys move visually.

EZ> A useless mode, if you ask me.

Agreed.  I was just reporting what occurred when I tried it out, for
everything I could think of which I had installed on this box and which
had -- or should have -- some bidi-awareness.

>> In gvim, both the h and l keys and the arrow keys move visually.

EZ> Please don't consider Vim a bidi-aware application.  All Vim does is
EZ> simple reversal of text on the screen.  It has no idea what to reverse
EZ> and what not, and neither does it support UAX#9.  It's small wonder it
EZ> can only support visual-order movement.

OK.  I knew that it could compose correctly and at first blush seemed
aware.  But again this was just reporting what occurrec when I tried
cursor movement w/in a r2l paragraph.

>> In Openoffice.org, the arrow keys move logically.

EZ> Again, please be more specific: what does the left arrow key do in a
EZ> R2L paragraph?  Does it go backwards in the text, eventually ending up
EZ> at the first text character, or does it go forward?

Again, I used the terms correctly.  If I wrote logically then the RIGHT
key moved forward in reading direction and the LEFT backward.

I will that OOo's cursor (a line at which the next char will insert)
changes when there is bidi text; a tail at its top points in the reading
direction of the surounding text.

Apologies if any of the above seems short....

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: arrow keys vs. C-f/b/n/p

[Eli Zaretskii]

> From: "Drew Adams" <drew.adams@oracle.com>
> Date: Fri, 11 Jun 2010 13:16:54 -0700
> Cc: 
> 
> Why should a change in text-insertion direction cause lots of directional
> keys/commands to flip (but not all, apparently)?

The directional keys do not flip.

> > If "forward" doesn't really mean forward in the text,
> 
> What about forward in the _buffer_?

It's the same thing.  "Forward" means in the order of increasing
character positions.

> Should `forward-char' reflect the R2L/L2R quality of the surrounding chars, or
> should it just reflect the buffer directionality (bob -> eob)?

The latter.  That is what logical order is all about.

> What about beginning of buffer? Should it sometimes become end of
> buffer?

No.

> Should that happen only if the entire buffer is R2L?

No.

> Anyone smell a can of worms around here?

No.

> But for the ordinary cursor movements, I still raise the naive question: Why not
> just let "forward" be toward eob?

It _is_ toward EOB.  And it should never be changed.

Re: arrow keys vs. C-f/b/n/p

[James Cloos]

>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:

EZ> I don't think so.  At least, we should remove gvim from this sample,
EZ> and wait for James to provide more accurate information about the rest.

The info I provided was an accurate and precise account of actually
testing each app with some arabic text which I had at hand.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: arrow keys vs. C-f/b/n/p

[James Cloos]

>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:

>> Oh, and my personal preference, as a l2r reader who occasionally deals
>> with r2l text, is for both C-f/C-b and LEFT/RIGHT to *always* move in
>> logical order.

EZ> That's what Emacs does now.  Both C-f/C-b and the arrows move in the
EZ> logical order.  There's no visual-order movement in Emacs (yet).

Yes.  My point was that I like the current behavior.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: arrow keys vs. C-f/b/n/p

[Eli Zaretskii]

> From: "Drew Adams" <drew.adams@oracle.com>
> Date: Fri, 11 Jun 2010 15:46:18 -0700
> Cc: emacs-devel@gnu.org
> 
> And if we go with a preconceived idea that C-f should change directions
> depending on the text surrounding the cursor (R2L vs L2R), then we are also
> likely to go wrong.

C-f does not change directions.  It always goes forward in the buffer.
The effect on the cursor position on the screen and direction of its
change is another matter.

> BTW, what do we do when point is at a boundary (R2L/L2R or L2R/R2L)?  Which way
> is forward?

Same as always: towards the next buffer position.

> I imagine that these questions are not new for R2L folks. Hasn't some sort of
> standard behavior been worked out for buffers/documents that combine R2L and L2R
> text? If so, how does that conventional behavior fit with Emacs?

What's currently implemented in Emacs is what this "prior art" does
and expects.

> Will `forward-char' be DWIMed to fit R2L

It shouldn't.  forward-char should always go to the next character in
buffer's logical order.  If we don't do that, most Emacs features will
stop working.

> Eventually someone who knows better will chime in and set things
> straight. ;-)

I hope I did.

Re: arrow keys vs. C-f/b/n/p

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Sat, 12 Jun 2010 11:34:39 +0200
> 
> I use left and right cursor keys (actually, all cursor keys) together
> with autorepeat for visual navigation: you press the key down, follow
> the movement, and stop when you are pretty much there.  Logical movement
> is not good for that.

Why not? it will get you to where you want, eventually.

Re: arrow keys vs. C-f/b/n/p

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Sat, 12 Jun 2010 12:17:05 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > In general, I believe the visual-mode movement is useful only to
> > people who don't really read the bidirectional text they traverse with
> > the cursor-motion keys.
> 
> I almost never use a single left/right cursor-motion keypress.  And when
> I traverse material with the cursor-motion keys, I don't read it _while_
> traversing.  I get glimpses, like when using autorepeat with
> scroll-up/scroll-down.  My autorepeat rate is about 20 characters per
> second.  At that speed, I can't really follow forward/backward jumps.
> 
> So no, I "don't really read the bidirectional text I traverse with the
> cursor-motion keys", and I want to see a single person either claim that
> she presses the cursor key once for every character she reads, or that
> she reads text exactly at autorepeat speed, including jumping backwards
> and forwards in mixed L2R/R2L text.

Then maybe a visual-mode motion is more important than I thought.  But
it still cannot be the only one, and I still think the logical-order
motion is more important, albeit perhaps for shorter sequences of
single-character movement.

Re: arrow keys vs. C-f/b/n/p

[Eli Zaretskii]

> From: Lennart Borgman <lennart.borgman@gmail.com>
> Date: Fri, 11 Jun 2010 20:04:28 +0200
> Cc: Eli Zaretskii <eliz@gnu.org>, David Kastrup <dak@gnu.org>,
> 	James Cloos <cloos@jhcloos.com>, emacs-devel@gnu.org
> 
> Looking around on the internet (not being a bidi reader) it seems to
> me that it is mostly a programming mistake to let the arrow keys
> change direction. (Somehow maybe also from a beginners perspective the
> right arrow might mean "go forward".) However for someone editing
> texts in some way this is awkward IMO. Here are some links:

I know all these references, and have a few more.  I agree that it
would be nice to have all the features they preach about.  People are
welcome to step forward and implement them; I even made a point of
implementing some infrastructure to support these features.

I'm not saying that logical-order motion is the only mode we should
have.  I'm saying that it's the absolute minimum, without which Emacs
would be unusable with bidirectional text.  And that is why I
implemented them -- to have Emacs minimally usable for editing
bidirectional scripts.

> - http://wiki.sibawayhi.org/keyboard - READING IS NOT CURSORING!

The author of this page is an enthusiastic users of the current Emacs,
so I guess practice is not necessarily so black-and-white as it
sounds.

> - http://www.qsm.co.il/Hebrew/logicUI22.htm - "The logical approach is
> sometimes preferred by implementors, mostly because its implementation
> is easier.

"Easier" my foot!  Believe me, it's not at all easy in Emacs, with all
its advanced display features.

Re: arrow keys vs. C-f/b/n/p

[Uday S Reddy]

Dear Eli, Thank you for a very clear post.  The crux of the matter seems to be 
contained the following two paragraphs:

>
> This is the reason why the<right>  arrow key moves like C-f in a L2R
> paragraph, and why the<left>  arrow key moves like C-f in a R2L
> paragraph.  (And similarly with C-b.)  It is so that Emacs behaves as
> expected in a purely R2L paragraph containing only R2L text.  Any
> other operation would be confusingly counter-intuitive: the<right>
> key would actually move cursor _to_the_left_!
>
> Again, we are talking _only_ about _logical-order_movement_!

So, you are saying that the <right> key moves right the majority of time, 
flipping direction only for the embedded text that switches direction.  When it 
switches direction for the embedded text, the <right> key would move left, 
"confusingly counter-intuitively" in your words!

The purist in me wants to say that the <right> key should always move right, 
but I think I can live with this compromise if it is known to be acceptable and 
useful to the bidirectional users.

Cheers,
Uday

Re: arrow keys vs. C-f/b/n/p

[Eli Zaretskii]

> From: Uday S Reddy <u.s.reddy@cs.bham.ac.uk>
> Date: Sat, 12 Jun 2010 12:13:23 +0100
> 
> So, you are saying that the <right> key moves right the majority of time, 
> flipping direction only for the embedded text that switches direction.

Yes.

> The purist in me wants to say that the <right> key should always move right, 

This is visual-mode motion, which we don't have yet.

Re: arrow keys vs. C-f/b/n/p

[Eli Zaretskii]

> From: James Cloos <cloos@jhcloos.com>
> Date: Sat, 12 Jun 2010 06:17:36 -0400
> Cc: cyd@stupidchicken.com, dak@gnu.org, emacs-devel@gnu.org
> 
> >> In Openoffice.org, the arrow keys move logically.
> 
> EZ> Again, please be more specific: what does the left arrow key do in a
> EZ> R2L paragraph?  Does it go backwards in the text, eventually ending up
> EZ> at the first text character, or does it go forward?
> 
> Again, I used the terms correctly.  If I wrote logically then the RIGHT
> key moved forward in reading direction and the LEFT backward.

So what does the RIGHT arrow do in a R2L paragraph, i.e. in a
paragraph that is displayed flushed all the way to the right margin of
the window?  Are you saying that the RIGHT arrow actually moves to the
_left_ in such a paragraph?  (You say that RIGHT moves in the reading
direction, but the reading direction in a R2L paragraph is right to
left.)

Re: arrow keys vs. C-f/b/n/p

[James Cloos]

>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:

EZ> So what does the RIGHT arrow do in a R2L paragraph, i.e. in a
EZ> paragraph that is displayed flushed all the way to the right margin of
EZ> the window?  Are you saying that the RIGHT arrow actually moves to the
EZ> _left_ in such a paragraph?  (You say that RIGHT moves in the reading
EZ> direction, but the reading direction in a R2L paragraph is right to
EZ> left.)

Yes, exactly.

I didn't try any of the tests in a Hebrew, Arabic or Farsi locale,
though, and a retest confirms, at least when running in en_US.UTF-8,
that I have to manually set the paragraph direction with the align
right paragraph control.

I just tried using LC_ALL=fa_FA.UTF-8.  With that the arrow keys do
exatly the opposite of what they do with LANG=en_US.UTF-8; the LEFT
key moves forward in logical order and the RIGHT key backwards.  But
they still always move in logical order, just swapping the forward vs
back meanings as compared to what they do when run in a l2r locale.
Weird!

Oh, and the default paragraph alignment does become Align Right,
as expected.

I still prefer only logical moves, with C-f moving forward, C-b moving
backward, LEFT always matching C-b and RIGHT always matching C-f.  But
I have no idea what users who use a r2e LANG/LC_* would want.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

RE: arrow keys vs. C-f/b/n/p

[Drew Adams]

Very clear post, which helps me understand. Thank you.

> Now please note an important detail: in a L2R paragraph, C-f generally
> moves cursor _to_the_right_, even though it could sometimes change
> direction and move to the left, when we are traversing R2L text
> embedded into a L2R paragraph.  Similarly, in a R2L paragraph, C-f
> generally moves cursor _to_the_left_.  In the important special case,
> when L2R paragraphs include only L2R text and R2L paragraphs include
> only R2L text, cursor motion with C-f is strictly to the right
> resp. to the left.

That sounds reasonable.

> This is the reason

What is the reason?

> why the <right> arrow key moves like C-f in a L2R
> paragraph, and why the <left> arrow key moves like C-f in a R2L
> paragraph.  (And similarly with C-b.)

FWIW, I don't see a supporting argument for why the arrows should have different
behavior from C-f/C-b.

> It is so that Emacs behaves as expected

Well, there's the reason, but what is it? What is "expected" of the arrow keys,
that means they should act differently from C-f/C-b?

You made it clear that we are, in all cases, talking only about logical order
(which you defined as buffer order, beginning to end). And both arrows and
C-f/C-b follow the logical order (a-z, in your example).

It is still a question why the paragraph direction should reverse which arrow
key moves forward vs backward (in logical order). You might have a good reason
for that, but so far I haven't seen it. You've only said that this is "expected"
of the arrows.

Don't get me wrong - I don't really care. I'm just pointing out that you haven't
really given a good reason for this, beyond some hand-waving ("expected").

> in a purely R2L paragraph containing only R2L text.  Any
> other operation would be confusingly counter-intuitive: the <right>
> key would actually move cursor _to_the_left_!

How is that different for C-f/C-b? Should one pair be counter-intuitive but not
the other? Why?

> Again, we are talking _only_ about _logical-order_movement_!

Great.

Again, thanks for a clear post. It really was helpful. 

And as I say, I don't really care one way or the other for myself (I'll probably
always use L2R text). But it might help if you do give some rationale for why
arrows, but not C-f/C-b, should behave "intuitively".

RE: arrow keys vs. C-f/b/n/p

[Drew Adams]

> > It just switches between "C-f = right and C-b = left" and
> > "C-f = left and C-b = right" based on the paragraph's direction.
> > Which seems eminently meaningful since the associating between
> > "forward" and "right" is just based on our usual convention of
> > writing L2R.
> 
> Exactly.

Exactly what? That just _restates_ the claim that the left arrow key should
follow the logical order like C-f and not follow the logical order like C-b, in
a R2L para.

No _reason_ has yet been given for such arrow key behavior. You've said it is
needed in order to not be "counter-intuitive". But that argument alone should
apply as well to C-f/C-b.

To be clear, I don't have a problem with the behavior you describe. But I don't
see any supporting argument for it either, so far.

RE: arrow keys vs. C-f/b/n/p

[Drew Adams]

> > On the surface this seems wrong and overly complicated (to 
> > me). "Eminently meaningful" it no doubt is, but it seems
> > somehow bass ackwards. ;-)
> 
> Do you still think that, after reading my explanations?

No, your explanations clarify things.

I think you still have not given a good reason for swapping the arrow keys in an
R2L para. But I have no problem with the behavior you describe, personally.

> It does seem wrong.  Which is why Emacs doesn't do that.  Do you
> agree, after reading my explanations?

Yes, see above.

> > Why would it be wrong for `right' (right arrow) to always 
> > represent movement toward the right, regardless of R2L/L2R?
> 
> It is not wrong, which is why Emacs behaves precisely like that.
> 
> The only exception is when R2L text is embedded in a L2R paragraph
> (i.e. in otherwise mostly L2R text).  Since arrow keys still move in
> logical order, the direction is reversed while we are traversing R2L
> text; it returns to its usual sense when point is back inside L2R
> text.

And that's true also for C-f/C-b, IIUC.  And it sounds good to me.

> Having the right arrow always go right (i.e. in the visual order)
> means that users will be unable to press the Shift key and have the
> text they traverse be marked correctly.  That is why we have only the
> logical-order movement for now -- because it is indispensable.  While
> I don't object to have a visual-order movement mode, I think it is
> much less useful, and it must be a separate mode in any case.

Sounds good.

> In general, I believe the visual-mode movement is useful only to
> people who don't really read the bidirectional text they traverse with
> the cursor-motion keys.  Any person who actually reads and understand
> the text will need the logical-order motion first and foremost, and
> the fact that direction is sometimes switched for a short period of
> time is expected and understood by users of bidirectional scripts.  It
> is a necessary complication, and one that is not hard to become used
> to -- because the cursor moves in the reading order, the same order
> our eyes move when we scan text.
> 
> At stake here is a very fundamental issue: the reading order is still
> from character position 1 to position 2 to 3 and so forth.  The order
> on the screen does not change this fundamental fact.  If you break
> this fundamental relation between the buffer order and the reading
> order, users will throw Emacs out the window.

I think I was trying to say the same thing (but with less understanding).

The meaning of "forward" in most command names should stay as it is, which is
the logical (buffer) order.  You've confirmed that that will be the case.

> > Just which things really need to be flipped/mirrored?
> 
> We need to break the one-to-one relation between buffer's logical
> order and the left/right direction.  This is what bidirectional text
> support is all about: "leftmost" no longer means "the first", because
> screen direction and screen sides no longer correspond to buffer
> order, in general.

That's fine.

> > Why not leave everything as it is wrt "forward" movement, 
> > and let users understand that "forward" always means toward
> > the right (in both L2R and R2L)?

I expressed myself wrong here. What I meant to say was "toward buffer end", as I
stated more clearly in other places. That was my concern: "forward" commands
should move in the direction of point-min toward point-max: buffer order.

And you've confirmed that that is the case. (For both C-f/C-b and arrows - the
only question being which arrow moves which way.)

> > Maybe you'll say that at least `C-d' and `backspace' need 
> > to flip direction (forward/backward)
> 
> They don't, and shouldn't, IMO.  C-d always deletes the character at
> point, and Backspace always deletes the character before point.  The
> basic Emacs notions of "before point" and "after point" are not
> changed by the bidi support, because doing that would mean that
> everything in Emacs would stop working.

Again, very good to hear. In general, what you've done fits what I would expect
naively.

> In general, the basic design of the bidi support changes only the
> display features. It changes nothing in how buffer text is processed
> and accessed.  This is done on purpose: I strongly believe that
> bidirectional text is a display-only feature.  It must not change
> anything else.

Music to my ignorant ears.  Just what I wanted to hear.

> There's no "left" and "right" in the buffer.  Insertion doesn't move
> point to the left, it moves point farther from the beginning of the
> buffer, i.e. it increases its buffer position.

Sehr gut.

> Whether this is right or left on the screen depends on the
> surrounding bidirectional context.
> 
> > If nothing else, perhaps the naive questions expressed here 
> > will help you document the bidi stuff for other naive users - HTH.
> 
> Naive users who read bidirectional scripts do not need this to be
> explained: they expect this behavior by default.

Maybe. Still, your explanation helped this naive user who does _not_ read bidi
to understand the feature. Something similar in the Elisp manual (a concise
statement from your first mail today) would help Emacs users generally, whether
or not they actually use bidi.

> > Perhaps something analogous to node `(elisp) Not Intervals' would be
> > helpful for this, explaining the logic behind the design choices.
> 
> We don't explain design choices in the manuals.  There're plenty of
> comments in the code to explain these decisions; if you or someone
> else can point out where more commentary is in order, I'd be more than
> happy to add whatever is missing, being a documentation maniac.

Just explain what you explained in your first mail today: logical order (whether
C-f/C-b or arrows), paragraphs that are only R2L or only L2R, etc. You said it
very clearly and fairly succinctly. I think it probably cleared things up for
several of us.

RE: arrow keys vs. C-f/b/n/p

[Drew Adams]

> EZ> That's what Emacs does now.  Both C-f/C-b and the arrows 
> EZ> move in the logical order.  There's no visual-order movement
> EZ> in Emacs (yet).
> 
> Yes.  My point was that I like the current behavior.

And as Eli described it it makes sense to me.  Two down...

RE: arrow keys vs. C-f/b/n/p

[Drew Adams]

> > Why not just let "forward" be toward eob?
> 
> It _is_ toward EOB.  And it should never be changed.

You got my vote, at least. ;-)

Re: arrow keys vs. C-f/b/n/p

[Lennart Borgman]

On Sat, Jun 12, 2010 at 11:27 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Lennart Borgman <lennart.borgman@gmail.com>
>> Date: Fri, 11 Jun 2010 19:09:44 +0200
>> Cc: Chong Yidong <cyd@stupidchicken.com>, Eli Zaretskii <eliz@gnu.org>, David Kastrup <dak@gnu.org>,
>>       emacs-devel@gnu.org
>>
>> On Fri, Jun 11, 2010 at 6:29 PM, James Cloos <cloos@jhcloos.com> wrote:
>> >
>> > In Openoffice.org, the arrow keys move logically.
>>
>> Seems like a mistake to me.
>
> What is the mistake?  What do you think James described by the above
> sentence?

I read it as meaning that the arrow keys does not always move
visually. (I.e. right arrow keys could move to the left.)

>> I think switching the visual directions of the arrow will interfere
>> badly because using them is most certainly in low level memory (body
>> memory) in the user.
>
> What program do you think switched the visual directions of the arrow
> keys?  Emacs doesn't do that: the <right> key moves to the right, and
> the <left> key moves to the left.  (They didn't, before I made the
> changes in the bindings of the arrow keys.)

Are you saying that the right arrow key etc will always move visually
to the right? In that case it is good.

Or are you saying that <right> key in Emacs now always does not mean
the right arrow key? That seems bad in my opinion.

Re: arrow keys vs. C-f/b/n/p

[Lennart Borgman]

On Sat, Jun 12, 2010 at 12:48 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>
> I know all these references, and have a few more.

I am sure you do.

> I agree that it
> would be nice to have all the features they preach about.  People are
> welcome to step forward and implement them;

I believe you have done a great job implementing this, but I do not
think it is finished until the visual movement is implemented too.

> I'm not saying that logical-order motion is the only mode we should
> have.  I'm saying that it's the absolute minimum, without which Emacs
> would be unusable with bidirectional text.

I think you are right. And I think that visual movement should be the default.

> And that is why I
> implemented them -- to have Emacs minimally usable for editing
> bidirectional scripts.

It is great.

>> - http://wiki.sibawayhi.org/keyboard - READING IS NOT CURSORING!
>
> The author of this page is an enthusiastic users of the current Emacs,
> so I guess practice is not necessarily so black-and-white as it
> sounds.

I do not know who it is, but I really agree with his arguing here.

>> - http://www.qsm.co.il/Hebrew/logicUI22.htm - "The logical approach is
>> sometimes preferred by implementors, mostly because its implementation
>> is easier.
>
> "Easier" my foot!  Believe me, it's not at all easy in Emacs, with all
> its advanced display features.

I am sure they only referred to the (missing) implementation of visual
movement ;-)

And the core part of this implementation is of course already done
when the text is displayed correctly on screen. (Which does not mean
that the rest is necessarily easy to do.)

Re: arrow keys vs. C-f/b/n/p

[Eli Zaretskii]

> From: "Drew Adams" <drew.adams@oracle.com>
> Date: Sat, 12 Jun 2010 07:26:53 -0700
> Cc: emacs-devel@gnu.org
> 
> I think you still have not given a good reason for swapping the arrow keys in an
> R2L para.

The reason for that is the important special case where the paragraph
is made of R2L text only.  In such a paragraph, moving to the left
means moving forward in the buffer.  That is why the left arrow does
the same as C-f in a R2L paragraph.

> Just explain what you explained in your first mail today: logical order (whether
> C-f/C-b or arrows), paragraphs that are only R2L or only L2R, etc. You said it
> very clearly and fairly succinctly. I think it probably cleared things up for
> several of us.

I explained some of that already, in the node "Bidirectional Editing"
in the Emacs user manual.  The bidi-aware behavior of the arrow keys
is described in the node "Moving Point".  Please see if that's enough.

Re: arrow keys vs. C-f/b/n/p

[Eli Zaretskii]

> From: Lennart Borgman <lennart.borgman@gmail.com>
> Date: Sat, 12 Jun 2010 16:49:25 +0200
> Cc: cyd@stupidchicken.com, dak@gnu.org, cloos@jhcloos.com, emacs-devel@gnu.org
> 
> On Sat, Jun 12, 2010 at 11:27 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> >> From: Lennart Borgman <lennart.borgman@gmail.com>
> >> Date: Fri, 11 Jun 2010 19:09:44 +0200
> >> Cc: Chong Yidong <cyd@stupidchicken.com>, Eli Zaretskii <eliz@gnu.org>, David Kastrup <dak@gnu.org>,
> >>       emacs-devel@gnu.org
> >>
> >> On Fri, Jun 11, 2010 at 6:29 PM, James Cloos <cloos@jhcloos.com> wrote:
> >> >
> >> > In Openoffice.org, the arrow keys move logically.
> >>
> >> Seems like a mistake to me.
> >
> > What is the mistake?  What do you think James described by the above
> > sentence?
> 
> I read it as meaning that the arrow keys does not always move
> visually. (I.e. right arrow keys could move to the left.)

Only when the bidirectional context changes, e.g. you traverse a R2L
text in a L2R paragraph.

> Are you saying that the right arrow key etc will always move visually
> to the right?

No.  Cursor motion works in the logical order, not in the visual
order.

Re: arrow keys vs. C-f/b/n/p

[Eli Zaretskii]

> From: "Drew Adams" <drew.adams@oracle.com>
> Date: Sat, 12 Jun 2010 07:26:42 -0700
> 
> > Now please note an important detail: in a L2R paragraph, C-f generally
> > moves cursor _to_the_right_, even though it could sometimes change
> > direction and move to the left, when we are traversing R2L text
> > embedded into a L2R paragraph.  Similarly, in a R2L paragraph, C-f
> > generally moves cursor _to_the_left_.  In the important special case,
> > when L2R paragraphs include only L2R text and R2L paragraphs include
> > only R2L text, cursor motion with C-f is strictly to the right
> > resp. to the left.
> 
> That sounds reasonable.
> 
> > This is the reason
> 
> What is the reason?

That important special case I described in the last sentence.

> > in a purely R2L paragraph containing only R2L text.  Any
> > other operation would be confusingly counter-intuitive: the <right>
> > key would actually move cursor _to_the_left_!
> 
> How is that different for C-f/C-b?

C-f/C-b do not have right/left connotation with them.

> Should one pair be counter-intuitive but not
> the other? Why?

Having C-f move to the left in R2L text is not counter-intuitive.  To
the left is forward in such text, so it matches intuition exactly.

Re: arrow keys vs. C-f/b/n/p

[Lennart Borgman]

On Sat, Jun 12, 2010 at 6:11 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Lennart Borgman <lennart.borgman@gmail.com>
>> Date: Sat, 12 Jun 2010 16:49:25 +0200
>> Cc: cyd@stupidchicken.com, dak@gnu.org, cloos@jhcloos.com, emacs-devel@gnu.org
>>
>> On Sat, Jun 12, 2010 at 11:27 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>> >> From: Lennart Borgman <lennart.borgman@gmail.com>
>> >> Date: Fri, 11 Jun 2010 19:09:44 +0200
>> >> Cc: Chong Yidong <cyd@stupidchicken.com>, Eli Zaretskii <eliz@gnu.org>, David Kastrup <dak@gnu.org>,
>> >>       emacs-devel@gnu.org
>> >>
>> >> On Fri, Jun 11, 2010 at 6:29 PM, James Cloos <cloos@jhcloos.com> wrote:
>> >> >
>> >> > In Openoffice.org, the arrow keys move logically.
>> >>
>> >> Seems like a mistake to me.
>> >
>> > What is the mistake?  What do you think James described by the above
>> > sentence?
>>
>> I read it as meaning that the arrow keys does not always move
>> visually. (I.e. right arrow keys could move to the left.)
>
> Only when the bidirectional context changes, e.g. you traverse a R2L
> text in a L2R paragraph.

This does not seem good to me.

>> Are you saying that the right arrow key etc will always move visually
>> to the right?
>
> No.  Cursor motion works in the logical order, not in the visual
> order.

I think the default should be that cursor keys (and keys like page
up/down) should move visually.

If someone want them to move logically that is ok and it should be
easy to fix if there are commands that move logically to rebind to.

Re: arrow keys vs. C-f/b/n/p

[Andreas Schwab]

In the standard KDE editor component the cursor keys always move in
logical order, with a small arrow at the cursor indicating the current
direction.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: arrow keys vs. C-f/b/n/p

[Andreas Schwab]

James Cloos <cloos@jhcloos.com> writes:

> In gedit C-f and C-b move forward and backward logically

By default, C-f and C-b in gedit don't do cursor motion.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Current head won't build

[Randal L. Schwartz]

This commit broke the build, via bisecting back to yesterday's good build:

Author: Juri Linkov <juri@jurta.org>
Date:   Fri Jun 11 21:51:00 2010 +0300

    Move version control related files to the "vc" subdirectory.
    * add-log.el, cvs-status.el, diff.el, diff-mode.el, ediff-diff.el,
    * ediff.el, ediff-help.el, ediff-hook.el, ediff-init.el,
    * ediff-merg.el, ediff-mult.el, ediff-ptch.el, ediff-util.el,
    * ediff-vers.el, ediff-wind.el, emerge.el, log-edit.el, log-view.el,
    * pcvs-defs.el, pcvs.el, pcvs-info.el, pcvs-parse.el, pcvs-util.el,
    * smerge-mode.el, vc-annotate.el, vc-arch.el, vc-bzr.el, vc-cvs.el,
    * vc-dav.el, vc-dir.el, vc-dispatcher.el, vc.el, vc-git.el,
    * vc-hg.el, vc-hooks.el, vc-mtn.el, vc-rcs.el, vc-sccs.el,
    vc-svn.el:
    Move files to the "vc" subdirectory.

-- 
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Smalltalk/Perl/Unix consulting, Technical writing, Comedy, etc. etc.
See http://methodsandmessages.vox.com/ for Smalltalk and Seaside discussion

RE: arrow keys vs. C-f/b/n/p

[Drew Adams]

> > I think you still have not given a good reason for swapping 
> > the arrow keys in an R2L para.
> 
> The reason for that is the important special case where the paragraph
> is made of R2L text only.  In such a paragraph, moving to the left
> means moving forward in the buffer.  That is why the left arrow does
> the same as C-f in a R2L paragraph.

I understand. I think that is open for debate, but you'll get no debate on it
from me.

I'll just say that it's not clear that this arrow reversal and its consequent
difference from C-f/C-b is a great idea, especially since it happens only if the
entire para is R2L. Sounds like these behavior differences (almost modal) could
be confusing. Probably bidi user experience will answer the question of whether
it is a good choice.

> > Just explain what you explained in your first mail today: 
> > logical order (whether C-f/C-b or arrows), paragraphs that
> > are only R2L or only L2R, etc. You said it very clearly and
> > fairly succinctly. I think it probably cleared things up for
> > several of us.
> 
> I explained some of that already, in the node "Bidirectional Editing"
> in the Emacs user manual.  The bidi-aware behavior of the arrow keys
> is described in the node "Moving Point".  Please see if that's enough.

Hm.  Well, you try to keep things at a user, as opposed to an
implementation/design, level, which is correct for the Emacs manual.  From that
point of view, saying `"logical" (or "reading") order' is OK.

But I think you should also make it clear (clearer) that this "logical" order
corresponds to buffer position, even if that might seem more
programmer-oriented. Emacs users sooner or later have an understanding of buffer
positions, so you might as well anchor the notion of "logical" order as being
buffer order.

In fact, you might drop "logical" altogether - the word itself doesn't really
help here. If you speak about the order of the chars in the buffer (as opposed
to how they appear) I think that will be clear.

And changing "character position" to "buffer position" would help. The former is
ambiguous, or at least it could be read in different ways. It is char position
in the buffer that is important. I would suggest emphasizing buffer
order/position.

(In any case, you do touch on implementation/design anyway, when you speak about
display position.)

What I got from your mail yesterday, which I think is clear there, is that (a)
the buffer text stays put, regardless of how it might be displayed (I already
supposed that), and (b) cursor movement always follows buffer order: forward
means toward eob; backward means toward bob.

It is (b) that is not so clear from the doc. That notion of "logical" movement
(movement along the buffer-position gradient) is important to understand. 

It's one thing to indicate that display order can differ from buffer order. It's
another thing to make clear that cursor motion always follows buffer order
(until you introduce visual movement to the mix). `forward-foobar' pretty much
always implies movement toward eob. This is important to understand, for both
interactive use and programming. And I don't think it comes across in the
current doc.

Admittedly, it can be tricky to talk about these things. But I found your mail
yesterday to be clearer than the current doc. In the doc you say things like
"the buffer is reordered for display", which is correct but which could also be
misunderstood. The chars are not reordered in the buffer - the buffer itself is
not reordered. It is just that the displayed order differs from the buffer order
(which does not change).

Yes, I'm splitting hairs, but it can help to try reading it in as confusing a
manner as possible, to see what difficulties others might have. ;-)  HTH.

BTW, maybe the node should be called Bidirectional Text (or Editing
Bidirectional Text) instead of Bidirectional Editing. We've all been doing
bidirectional (multidirectional) editing forever.

Re: Current head won't build

[Chong Yidong]

merlyn@stonehenge.com (Randal L. Schwartz) writes:

> This commit broke the build, via bisecting back to yesterday's good build:
>
> Author: Juri Linkov <juri@jurta.org>
> Date:   Fri Jun 11 21:51:00 2010 +0300
>
>     Move version control related files to the "vc" subdirectory.

You'll need to `make bootstrap'.

Re: arrow keys vs. C-f/b/n/p

[Lennart Borgman]

On Sat, Jun 12, 2010 at 7:12 PM, Drew Adams <drew.adams@oracle.com> wrote:
>
> In fact, you might drop "logical" altogether - the word itself doesn't really
> help here. If you speak about the order of the chars in the buffer (as opposed
> to how they appear) I think that will be clear.


From what I read it seems to me that "logical" should mean "human
reading order", not the buffer character order.

I guess they are the same here, but that is an implementation detail
(which I think is good since it seems to simplify many things).

Re: arrow keys vs. C-f/b/n/p

[Uday S Reddy]

"Drew Adams" <drew.adams@oracle.com> writes:

> But I think you should also make it clear (clearer) that this
> "logical" order corresponds to buffer position, even if that might
> seem more programmer-oriented. Emacs users sooner or later have an
> understanding of buffer positions, so you might as well anchor the
> notion of "logical" order as being buffer order.

Drew, I am not sure why you think this buffer order is so fundamental.
An Emacs user doesn't see a buffer order.  There is an order in which
text is layed out on the screen.  There is an order in which the
language flows in the text.  These are the things that matter.  How the
characters are stored inside the Emacs buffer is an internal matter.

If an Emacs user graduates to becoming a programmer then he/she will
figure out how things work internally.  But it doesn't seem right to me
that the internal storage should dictate everybody's view of the world.

I notice that Lennart makes the same point later in the thread.

In fact, I began to write a response to you last night.  When I checked
things in the morning, there were comprehensive explanations from Eli,
which I thought settled the issues quite satisfactorily.

There is no harm in the users knowing that the buffer order is the same
as the logical, or reading, order.  But, even if the buffer order
happened to be different, I don't think the design issues would be any
different.

I am quite clear that the users should be given a clear conceptual model
that fits with their experience, not how things are implemented
internally. 

Cheers,
Uday

Re: arrow keys vs. C-f/b/n/p

[Eli Zaretskii]

> From: Lennart Borgman <lennart.borgman@gmail.com>
> Date: Sat, 12 Jun 2010 18:18:18 +0200
> Cc: cyd@stupidchicken.com, dak@gnu.org, cloos@jhcloos.com, emacs-devel@gnu.org
> 
> I think the default should be that cursor keys (and keys like page
> up/down) should move visually.

You are talking as a typical L2R user, as in that case visual and
logical order are identical.

For users of R2L scripts, what you suggest is a disaster.  If cursor
motion is visual by default, one cannot even mark text by holding
Shift and moving point with the arrow keys.  And that's just the tip
of the iceberg.  Without logical-order motion keys, Emacs (and every
other editor of similar sophistication) is much less useful.

Re: arrow keys vs. C-f/b/n/p

[Eli Zaretskii]

> From: Lennart Borgman <lennart.borgman@gmail.com>
> Date: Sat, 12 Jun 2010 19:25:41 +0200
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> 
> On Sat, Jun 12, 2010 at 7:12 PM, Drew Adams <drew.adams@oracle.com> wrote:
> >
> > In fact, you might drop "logical" altogether - the word itself doesn't really
> > help here. If you speak about the order of the chars in the buffer (as opposed
> > to how they appear) I think that will be clear.
> 
> 
> From what I read it seems to me that "logical" should mean "human
> reading order", not the buffer character order.

These two orders are identical.  The reason is that people type text
in the order they read it.  "Logical order" just means we store text
in the same order as it is read and typed.

> I guess they are the same here, but that is an implementation detail

No, it's not an implementation detail.  Emacs stores text in files,
and those files also use the logical order.  As soon as text is saved
on a file, the order of characters is not longer internal to Emacs.

In using logical order, Emacs follows the other applications; doing
that the other way around would mean the files created by Emacs are
incompatible with other applications.

Re: arrow keys vs. C-f/b/n/p

[Lennart Borgman]

On Sat, Jun 12, 2010 at 8:38 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Lennart Borgman <lennart.borgman@gmail.com>
>> Date: Sat, 12 Jun 2010 19:25:41 +0200
>> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
>>
>> On Sat, Jun 12, 2010 at 7:12 PM, Drew Adams <drew.adams@oracle.com> wrote:
>> >
>> > In fact, you might drop "logical" altogether - the word itself doesn't really
>> > help here. If you speak about the order of the chars in the buffer (as opposed
>> > to how they appear) I think that will be clear.
>>
>>
>> From what I read it seems to me that "logical" should mean "human
>> reading order", not the buffer character order.
>
> These two orders are identical.  The reason is that people type text
> in the order they read it.  "Logical order" just means we store text
> in the same order as it is read and typed.
>
>> I guess they are the same here, but that is an implementation detail
>
> No, it's not an implementation detail.  Emacs stores text in files,
> and those files also use the logical order.  As soon as text is saved
> on a file, the order of characters is not longer internal to Emacs.

Is not this is precisely what makes it an implementation detail?

> In using logical order, Emacs follows the other applications; doing
> that the other way around would mean the files created by Emacs are
> incompatible with other applications.

Re: arrow keys vs. C-f/b/n/p

[Eli Zaretskii]

> From: "Drew Adams" <drew.adams@oracle.com>
> Date: Sat, 12 Jun 2010 10:12:04 -0700
> Cc: emacs-devel@gnu.org
> 
> But I think you should also make it clear (clearer) that this "logical" order
> corresponds to buffer position

I did:

       Emacs stores right-to-left and bidirectional text in the so-called
    "logical" (or "reading") order: the buffer or string position of the
    first character you read precedes that of the next character.

> In fact, you might drop "logical" altogether - the word itself doesn't really
> help here.

This term is widely used in literature about editing bidirectional
text.

> And changing "character position" to "buffer position" would help.

See above: the text does talk about buffer positions.

> What I got from your mail yesterday, which I think is clear there, is that (a)
> the buffer text stays put, regardless of how it might be displayed (I already
> supposed that), and (b) cursor movement always follows buffer order: forward
> means toward eob; backward means toward bob.
> 
> It is (b) that is not so clear from the doc. That notion of "logical" movement
> (movement along the buffer-position gradient) is important to understand. 

The manual does touch on that:

       Because characters are reordered for display, Emacs commands that
    operate in the logical order or on stretches of buffer positions may
    produce unusual effects.  For example, `C-f' and `C-b' commands move
    point in the logical order, so the cursor will sometimes jump when
    point traverses reordered bidirectional text.  Similarly, a highlighted
    region covering a contiguous range of character positions may look
    discontinuous if the region spans reordered text.  This is normal and
    similar to behavior of other programs that support bidirectional text.

> Admittedly, it can be tricky to talk about these things. But I found your mail
> yesterday to be clearer than the current doc. In the doc you say things like
> "the buffer is reordered for display", which is correct but which could also be
> misunderstood. The chars are not reordered in the buffer - the buffer itself is
> not reordered. It is just that the displayed order differs from the buffer order
> (which does not change).

I see no "buffer is reordered" in the manual.  I see these instead:

    Reordering of bidirectional text into the "visual" order happens at
    display time.

    [...]

       The buffer-local variable `bidi-display-reordering' controls whether
    text in the buffer is reordered for display.

    [...]

       Because characters are reordered for display,

> BTW, maybe the node should be called Bidirectional Text (or Editing
> Bidirectional Text) instead of Bidirectional Editing. We've all been doing
> bidirectional (multidirectional) editing forever.

I will rename the section, thanks.

Re: Current head won't build

[Randal L. Schwartz]

>>>>> "Chong" == Chong Yidong <cyd@stupidchicken.com> writes:

Chong> merlyn@stonehenge.com (Randal L. Schwartz) writes:
>> This commit broke the build, via bisecting back to yesterday's good build:
>> 
>> Author: Juri Linkov <juri@jurta.org>
>> Date:   Fri Jun 11 21:51:00 2010 +0300
>> 
>> Move version control related files to the "vc" subdirectory.

Chong> You'll need to `make bootstrap'.

Did that.  Got this:

    Loading image...
    Loading international/fontset...
    Loading dnd...
    Loading tool-bar...
    Loading emacs-lisp/easymenu...
    Loading term/ns-win...
    Loading mwheel...
    Loading emacs-lisp/float-sup...
    Loading vc/vc-hooks.el (source)...
    (require cl) while preparing to dump
    make[2]: *** [emacs] Error 1
    make[1]: *** [src] Error 2
    make: *** [bootstrap] Error 2

-- 
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Smalltalk/Perl/Unix consulting, Technical writing, Comedy, etc. etc.
See http://methodsandmessages.vox.com/ for Smalltalk and Seaside discussion

Re: arrow keys vs. C-f/b/n/p

[Lennart Borgman]

On Sat, Jun 12, 2010 at 8:33 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Lennart Borgman <lennart.borgman@gmail.com>
>> Date: Sat, 12 Jun 2010 18:18:18 +0200
>> Cc: cyd@stupidchicken.com, dak@gnu.org, cloos@jhcloos.com, emacs-devel@gnu.org
>>
>> I think the default should be that cursor keys (and keys like page
>> up/down) should move visually.
>
> You are talking as a typical L2R user, as in that case visual and
> logical order are identical.


I do not think visual motion actually has anything to do with that.


> For users of R2L scripts, what you suggest is a disaster.  If cursor
> motion is visual by default, one cannot even mark text by holding
> Shift and moving point with the arrow keys.


I see no reason for this. The arrow keys will with visual motion move
just as they do if all the text where just L2R. So the visual part
will work as before (on the user side, implementation may have to
change).

Converting the visual region you visually see on the screen to a
logical range is a bit difficult, but not impossible.

The difficulty is of course to decide what the range on the screen
will be if the end points of the visual region happen to disagree
about the direction. (If there are more difficulties then I am unaware
of them at the moment.)

The answer to this is as far as I can see that the visual region in
this case no longer internally corresponds to a single range, but to
two noncontinuous ranges in the buffer. If I am correct on this, is
not this then a difficulty that must be handled to finish the bidi
support?


> And that's just the tip
> of the iceberg.  Without logical-order motion keys, Emacs (and every
> other editor of similar sophistication) is much less useful.

I think you are referring to the difficulty I suggested above. If not,
what more problems do you see?

RE: arrow keys vs. C-f/b/n/p

[Drew Adams]

> > But I think you should also make it clear (clearer) that this
> > "logical" order corresponds to buffer position, even if that might
> > seem more programmer-oriented. Emacs users sooner or later have an
> > understanding of buffer positions, so you might as well anchor the
> > notion of "logical" order as being buffer order.
> 
> Drew, I am not sure why you think this buffer order is so fundamental.
> An Emacs user doesn't see a buffer order.  There is an order in which
> text is layed out on the screen.  There is an order in which the
> language flows in the text.  These are the things that 
> matter.  How the characters are stored inside the Emacs buffer is
> an internal matter.
> 
> If an Emacs user graduates to becoming a programmer then he/she will
> figure out how things work internally.  But it doesn't seem 
> right to me that the internal storage should dictate everybody's
> view of the world.
> 
> I notice that Lennart makes the same point later in the thread.
> 
> In fact, I began to write a response to you last night.  When 
> I checked things in the morning, there were comprehensive
> explanations from Eli, which I thought settled the issues
> quite satisfactorily.
> 
> There is no harm in the users knowing that the buffer order 
> is the same as the logical, or reading, order.  But, even
> if the buffer order happened to be different, I don't think
> the design issues would be any different.
> 
> I am quite clear that the users should be given a clear 
> conceptual model that fits with their experience, not how
> things are implemented internally. 

It's not about "internal" implementation. It is, as you say, about a user's
conceptual model.  Is `point' or character position "internal"?

In Emacs, buffer order is part of a user model.  Calling it "reading order" is
fine, and can help, as long as the two notions correspond.  Calling it the
"logical" order doesn't really help.  (Is a "visual" order illogical? is it
physical?)  But as Eli has pointed out, that is apparently the term used in the
bidi literature.  Calling it "character" order begs the question: wrt what?

What's important is to explain what in fact it is, regardless of what you call
it.  What's the behavior?

What reading/logical/buffer order means includes, in particular, how movement
commands behave.  What I was concerned about is the behavior of typical
`forward-*' commands.  That behavior is part of a user's conceptual model.  (Not
to mention part of a programmer's mental model.  And in Emacs, users are also
often Elisp programmers.)

Anyway, it sounds like we are both happy with Eli's explanation and the design.
Whether bidi users will also need/want some alternative (additional) kind of
movement that respects "visual" order in some way is another question, which I
can't speak to.  I'm not a bidi user.

I'm happy knowing that `forward-*' means forward through the buffer, from
point-min toward point-max.  Phrase that fact anyway you like, but it should be
communicated to users.

RE: arrow keys vs. C-f/b/n/p

[Drew Adams]

> > But I think you should also make it clear (clearer) that 
> > this "logical" order corresponds to buffer position
> 
> I did:
>        Emacs stores right-to-left and bidirectional text in 
>     the so-called "logical" (or "reading") order: the buffer
>     or string position of the first character you read precedes
>     that of the next character.

Good.  My point was to make clear also the behavior of movement (e.g. typical
`forward-*' commands), not just the order of storage.  Forward movement follows
the storage order (typically).  That's worth saying.

> > In fact, you might drop "logical" altogether - the word 
> > itself doesn't really help here.
> 
> This term is widely used in literature about editing bidirectional
> text.

I see.  Then keep it, for readers familiar with that literature.  For those who
might not be familiar, "reading" order is perhaps the most helpful.  And it
helps to point out that this is the same as the buffer (storage) order that you
mentioned.

> > And changing "character position" to "buffer position" would help.
> 
> See above: the text does talk about buffer positions.

That text is not a problem.  I was referring to a passage where you spoke of
"character position" or character order.  That's less clear: character position
wrt what?  Answer: position in the buffer; position in reading order.

> > What I got from your mail yesterday, which I think is clear 
> > there, is that (a) the buffer text stays put, regardless of
> > how it might be displayed (I already supposed that), and
> > (b) cursor movement always follows buffer order: forward
> > means toward eob; backward means toward bob.
> > 
> > It is (b) that is not so clear from the doc. That notion of 
> > "logical" movement (movement along the buffer-position gradient)
> > is important to understand. 
> 
> The manual does touch on that:
> 
>        Because characters are reordered for display, Emacs 
>     commands that operate in the logical order or on stretches
>     of buffer positions may produce unusual effects.  For
>     example, `C-f' and `C-b' commands move point in the logical
>     order, so the cursor will sometimes jump when point
>     traverses reordered bidirectional text.  Similarly, 
>     a highlighted region covering a contiguous range of
>     character positions may look discontinuous if the region
>     spans reordered text.  This is normal and
>     similar to behavior of other programs that support 
>     bidirectional text.

Yes, I read that.  It's fine as far as it goes, but it stresses clearing up some
particular behavior (gotcha) that users might find confusing.  It mentions `C-f'
as a particular _example_, to illustrate the point.  But I'm not sure the
general point is communicated as such - it might be lost in the detailed
description of the gotcha.

This is not only about some particular commands that might happen to move point
in the logical order.  The important point that I think isn't brought out enough
is that one's existing notion of "forward" movement, and the typical `forward-*'
commands that one is used to, all act the same as does that `C-f' example.  They
act the same as they always have: forward typically means (continues to mean)
_toward eob_, even in a bidi setting.  That's not obvious.

> > Admittedly, it can be tricky to talk about these things. 
> > But I found your mail yesterday to be clearer than the
> > current doc. In the doc you say things like
> > "the buffer is reordered for display", which is correct but 
> > which could also be misunderstood. The chars are not
> > reordered in the buffer - the buffer itself is
> > not reordered. It is just that the displayed order differs 
> > from the buffer order (which does not change).
> 
> I see no "buffer is reordered" in the manual.

Hm.  That was a direct quote, from GNU Emacs 24.0.50.1 (i386-mingw-nt5.1.2600)
of 2010-06-07 on 3249CTO. Emacs manual, node Bidirectional Editing:

  "The buffer-local variable `bidi-display-reordering'
   controls whether text in the buffer is reordered for display."

Yes, I admitted I was splitting hairs.  Do as you like with the feedback.  You
are welcome to ignore it.

> I see these instead:
...
>        The buffer-local variable `bidi-display-reordering' 
>     controls whether text in the buffer is reordered for display.

So you do `see "buffer is reordered" in the manual', after all. ;-)

> > BTW, maybe the node should be called Bidirectional Text (or Editing
> > Bidirectional Text) instead of Bidirectional Editing. We've 
> > all been doing bidirectional (multidirectional) editing forever.
> 
> I will rename the section, thanks.

Thanks for your patience.

Re: arrow keys vs. C-f/b/n/p

[Lennart Borgman]

On Sat, Jun 12, 2010 at 8:58 PM, Lennart Borgman
<lennart.borgman@gmail.com> wrote:
> On Sat, Jun 12, 2010 at 8:33 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>>> From: Lennart Borgman <lennart.borgman@gmail.com>
>>> Date: Sat, 12 Jun 2010 18:18:18 +0200
>>> Cc: cyd@stupidchicken.com, dak@gnu.org, cloos@jhcloos.com, emacs-devel@gnu.org
>>>
>>> I think the default should be that cursor keys (and keys like page
>>> up/down) should move visually.
>>
>> You are talking as a typical L2R user, as in that case visual and
>> logical order are identical.
>
>
> I do not think visual motion actually has anything to do with that.
>
>
>> For users of R2L scripts, what you suggest is a disaster.  If cursor
>> motion is visual by default, one cannot even mark text by holding
>> Shift and moving point with the arrow keys.
>
>
> I see no reason for this. The arrow keys will with visual motion move
> just as they do if all the text where just L2R. So the visual part
> will work as before (on the user side, implementation may have to
> change).
>
> Converting the visual region you visually see on the screen to a
> logical range is a bit difficult, but not impossible.
>
> The difficulty is of course to decide what the range on the screen
> will be if the end points of the visual region happen to disagree
> about the direction. (If there are more difficulties then I am unaware
> of them at the moment.)
>
> The answer to this is as far as I can see that the visual region in
> this case no longer internally corresponds to a single range, but to
> two noncontinuous ranges in the buffer. If I am correct on this, is
> not this then a difficulty that must be handled to finish the bidi
> support?

I just tested in etc/HELLO to see how you have handled this. You took
another route than I expected and I think it makes sense.

When selecting a region with the two end points in parts with
different directions you instead split the visual region on screen.
(If anyone wonders just test in etc/HELLO.)

This is a nice way out of the problems implementing what I suggested
above. And I can see it makes sense too.


However it has nothing at all to do with the visual movements when
using the arrow keys. That movement can (and in my opinion should)
still be visual.


>> And that's just the tip
>> of the iceberg.  Without logical-order motion keys, Emacs (and every
>> other editor of similar sophistication) is much less useful.
>
> I think you are referring to the difficulty I suggested above. If not,
> what more problems do you see?
>

Re: arrow keys vs. C-f/b/n/p

[Eli Zaretskii]

> From: Lennart Borgman <lennart.borgman@gmail.com>
> Date: Sat, 12 Jun 2010 20:49:04 +0200
> Cc: drew.adams@oracle.com, emacs-devel@gnu.org
> 
> >> From what I read it seems to me that "logical" should mean "human
> >> reading order", not the buffer character order.
> >
> > These two orders are identical.  The reason is that people type text
> > in the order they read it.  "Logical order" just means we store text
> > in the same order as it is read and typed.
> >
> >> I guess they are the same here, but that is an implementation detail
> >
> > No, it's not an implementation detail.  Emacs stores text in files,
> > and those files also use the logical order.  As soon as text is saved
> > on a file, the order of characters is not longer internal to Emacs.
> 
> Is not this is precisely what makes it an implementation detail?

No.

Re: arrow keys vs. C-f/b/n/p

[Lennart Borgman]

On Sat, Jun 12, 2010 at 9:38 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> >
>> > No, it's not an implementation detail.  Emacs stores text in files,
>> > and those files also use the logical order.  As soon as text is saved
>> > on a file, the order of characters is not longer internal to Emacs.
>>
>> Is not this is precisely what makes it an implementation detail?
>
> No.

I see what you mean. For plain text files which Emacs handles this is true.

This was not what I meant.

Re: Current head won't build

[Juri Linkov]

>     Loading image...
>     Loading international/fontset...
>     Loading dnd...
>     Loading tool-bar...
>     Loading emacs-lisp/easymenu...
>     Loading term/ns-win...
>     Loading mwheel...
>     Loading emacs-lisp/float-sup...
>     Loading vc/vc-hooks.el (source)...
>     (require cl) while preparing to dump
>     make[2]: *** [emacs] Error 1
>     make[1]: *** [src] Error 2
>     make: *** [bootstrap] Error 2

This is strange.  Shouldn't all these files load from source?
They should be compiled later than this stage, but it seems
they are already compiled (except vc-hooks.el).

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: arrow keys vs. C-f/b/n/p

[Eli Zaretskii]

> From: Lennart Borgman <lennart.borgman@gmail.com>
> Date: Sat, 12 Jun 2010 20:58:18 +0200
> Cc: emacs-devel@gnu.org
> 
> > For users of R2L scripts, what you suggest is a disaster.  If cursor
> > motion is visual by default, one cannot even mark text by holding
> > Shift and moving point with the arrow keys.
> 
> 
> I see no reason for this. The arrow keys will with visual motion move
> just as they do if all the text where just L2R. So the visual part
> will work as before (on the user side, implementation may have to
> change).

It's not just an implementation issue.  Here's an example.  A text in
the buffer is

  abcde ABCDE xyz

It will be displayed as 

  abcde EDCBA xyz

Now, suppose I move cursor in visual order, as you suggest, holding
the Shift key, starting at the first character `a' and ending at `C'.
The underline below shows which characters I traversed:

  abcde EDCBA xyz
  ---------

Now, what characters do you think should be displayed with the region
face?

Now I type "M-w".  What text will that copy to the kill ring?

> Converting the visual region you visually see on the screen to a
> logical range is a bit difficult, but not impossible.

It's impossible.  There's no general algorithm to reverse the
logical-to-visual reordering of text, i.e. produce the original
logical-order text from its visual-order replica on the screen.  Only
relatively simple cases, without embeddings that push the
bidirectional level beyond 2 or 3 (UAX#9 allows up to 60 levels) can
be reversed.

> The difficulty is of course to decide what the range on the screen
> will be if the end points of the visual region happen to disagree
> about the direction.

That's the difficulty; what's your solution?  I don't know of any
solution that would make sense and would not complicate Emacs beyond
any reason.

> The answer to this is as far as I can see that the visual region in
> this case no longer internally corresponds to a single range, but to
> two noncontinuous ranges in the buffer. If I am correct on this, is
> not this then a difficulty that must be handled to finish the bidi
> support?

Do you have _any_ idea what your suggestion means?  Having a region
with several non-contiguous ranges of text blows up so many Emacs
features that I don't even know where to start explaining how wrong
that is.

> > And that's just the tip
> > of the iceberg.  Without logical-order motion keys, Emacs (and every
> > other editor of similar sophistication) is much less useful.
> 
> I think you are referring to the difficulty I suggested above. If not,
> what more problems do you see?

Text properties and overlays all depend on contiguous regions of text
in the logical order.  What you suggest means a complete rewrite of
all of them.  Some "difficulty"!

Re: arrow keys vs. C-f/b/n/p

[Eli Zaretskii]

> From: Lennart Borgman <lennart.borgman@gmail.com>
> Date: Sat, 12 Jun 2010 21:34:35 +0200
> Cc: emacs-devel@gnu.org
> 
> > The answer to this is as far as I can see that the visual region in
> > this case no longer internally corresponds to a single range, but to
> > two noncontinuous ranges in the buffer. If I am correct on this, is
> > not this then a difficulty that must be handled to finish the bidi
> > support?
> 
> I just tested in etc/HELLO to see how you have handled this. You took
> another route than I expected and I think it makes sense.

That other route is simply logical-order region, supported by a
logical-order cursor motion.

> When selecting a region with the two end points in parts with
> different directions you instead split the visual region on screen.

The important part is that the logical-order region is contiguous.
The seeming ``split'' on the screen is just the effect produced by
reordering characters for display.

> However it has nothing at all to do with the visual movements when
> using the arrow keys. That movement can (and in my opinion should)
> still be visual.

See my question in my other mail: with visual-order movement, the
region highlight and other similar features will behave
non-sensically.

Re: arrow keys vs. C-f/b/n/p

[Lennart Borgman]

On Sat, Jun 12, 2010 at 9:55 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Lennart Borgman <lennart.borgman@gmail.com>
>> Date: Sat, 12 Jun 2010 20:58:18 +0200
>> Cc: emacs-devel@gnu.org
>>
>> > For users of R2L scripts, what you suggest is a disaster.  If cursor
>> > motion is visual by default, one cannot even mark text by holding
>> > Shift and moving point with the arrow keys.
>>
>>
>> I see no reason for this. The arrow keys will with visual motion move
>> just as they do if all the text where just L2R. So the visual part
>> will work as before (on the user side, implementation may have to
>> change).
>
> It's not just an implementation issue.  Here's an example.  A text in
> the buffer is
>
>  abcde ABCDE xyz
>
> It will be displayed as
>
>  abcde EDCBA xyz
>
> Now, suppose I move cursor in visual order, as you suggest, holding
> the Shift key, starting at the first character `a' and ending at `C'.
> The underline below shows which characters I traversed:
>
>  abcde EDCBA xyz
>  ---------
>
> Now, what characters do you think should be displayed with the region
> face?


As I said in another message it would be "abcde CBA", i.e. exactly the
visual region that you now get when moving with the arrow keys so that
you get to C.

It has nothing to do with HOW you moved to C.


> Now I type "M-w".  What text will that copy to the kill ring?


See above.


>> Converting the visual region you visually see on the screen to a
>> logical range is a bit difficult, but not impossible.
>
> It's impossible.  There's no general algorithm to reverse the
> logical-to-visual reordering of text, i.e. produce the original
> logical-order text from its visual-order replica on the screen.  Only
> relatively simple cases, without embeddings that push the
> bidirectional level beyond 2 or 3 (UAX#9 allows up to 60 levels) can
> be reversed.


You mean bidirectional in bidirectional in bidirectional ...?

Yes, you are right. It would be impossible to find a continuous range for that.

But this still has nothing to do with the visual movement of the arrow keys.

Re: Current head won't build

[Randal L. Schwartz]

>>>>> "Juri" == Juri Linkov <juri@jurta.org> writes:

Juri> This is strange.  Shouldn't all these files load from source?
Juri> They should be compiled later than this stage, but it seems
Juri> they are already compiled (except vc-hooks.el).

Maybe "make bootstrap" should do some sort of "nearly-distclean"
that doesn't wipe out the configuration file I just generated?

-- 
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Smalltalk/Perl/Unix consulting, Technical writing, Comedy, etc. etc.
See http://methodsandmessages.vox.com/ for Smalltalk and Seaside discussion

Re: arrow keys vs. C-f/b/n/p

[Lennart Borgman]

On Sat, Jun 12, 2010 at 9:58 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>
>> When selecting a region with the two end points in parts with
>> different directions you instead split the visual region on screen.
>
> The important part is that the logical-order region is contiguous.
> The seeming ``split'' on the screen is just the effect produced by
> reordering characters for display.
>
>> However it has nothing at all to do with the visual movements when
>> using the arrow keys. That movement can (and in my opinion should)
>> still be visual.
>
> See my question in my other mail: with visual-order movement, the
> region highlight and other similar features will behave
> non-sensically.


So this is perhaps the most essential part of the argument  for not
having the arrow keys moving visually.

As I said I think your solution with one internal region split
visually on screen is the best solution. You have convinced me of
that.

I can now see the argument better for not letting the arrow keys move
visually, but I think I still disagree on that part. The reason I
disagree is the same as I stated from the beginning and that was
explained on the web page that said READING IS NOT CURSORING. Moving
the cursor depends on lower level memory in our brains.

 And of course: without letting the arrow keys move visually on the
screen there is no easy way at all to move visually on the screen.

This is actually a bit fun. Try creating with only blanks, just
spaces, tabs and newlines. Set some of the those chars to arabic and
let some be latin. Now try moving around the page using only the
visual keys. Try to go to the end of the page. (I suggest this as an
alternative to sudoku...)


As I said before: extending the region has nothing in per se with
letting the arrow keys move visually. The arrow keys is instead an
established way to move visually. And that is needed sometimes.

Re: arrow keys vs. C-f/b/n/p

[Stefan Monnier]

> To be clear, I don't have a problem with the behavior you
> describe.  But I don't see any supporting argument for it either,
> so far.

The argument for it has been given many times in this thread already.
So please re-read the thread until you understand it.


        Stefan

Re: arrow keys vs. C-f/b/n/p

[Stefan Monnier]

> Again, I used the terms correctly.  If I wrote logically then the RIGHT
> key moved forward in reading direction and the LEFT backward.

The difference between logical and visual movement only really makes
sense when you have R2L text within an L2R paragraph (or vice-versa).


        Stefan

Re: arrow keys vs. C-f/b/n/p

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>> Date: Sat, 12 Jun 2010 11:34:39 +0200
>> 
>> I use left and right cursor keys (actually, all cursor keys) together
>> with autorepeat for visual navigation: you press the key down, follow
>> the movement, and stop when you are pretty much there.  Logical movement
>> is not good for that.
>
> Why not? it will get you to where you want, eventually.

Because the cursor does not just flow, but also jumps.

-- 
David Kastrup

Re: arrow keys vs. C-f/b/n/p

[Eli Zaretskii]

> From: Lennart Borgman <lennart.borgman@gmail.com>
> Date: Sat, 12 Jun 2010 22:07:36 +0200
> Cc: emacs-devel@gnu.org
> 
> >  abcde EDCBA xyz
> >  ---------
> >
> > Now, what characters do you think should be displayed with the region
> > face?
> 
> 
> As I said in another message it would be "abcde CBA", i.e. exactly the
> visual region that you now get when moving with the arrow keys so that
> you get to C.

But that doesn't make sense: the cursor traverses characters that you
don't paint in the region face, and the region face is put on
characters the cursor didn't move through.

Re: arrow keys vs. C-f/b/n/p

[Eli Zaretskii]

> From: Lennart Borgman <lennart.borgman@gmail.com>
> Date: Sat, 12 Jun 2010 22:26:29 +0200
> Cc: emacs-devel@gnu.org
> 
> I can now see the argument better for not letting the arrow keys move
> visually, but I think I still disagree on that part. The reason I
> disagree is the same as I stated from the beginning and that was
> explained on the web page that said READING IS NOT CURSORING. Moving
> the cursor depends on lower level memory in our brains.

Yes, but us users of R2L scripts have that memory augmented, so we
don't have problem with the current behavior.

>  And of course: without letting the arrow keys move visually on the
> screen there is no easy way at all to move visually on the screen.

I see nothing non-easy with logical-order motion.

> This is actually a bit fun. Try creating with only blanks, just
> spaces, tabs and newlines. Set some of the those chars to arabic and
> let some be latin. Now try moving around the page using only the
> visual keys. Try to go to the end of the page. (I suggest this as an
> alternative to sudoku...)

You can mock it as much as you want, but users of R2L scripts do this
every day, and have no problem at all.

Anyway, we are beginning to repeat ourselves.  It's clear we disagree,
so let's leave it at that.

Re: arrow keys vs. C-f/b/n/p

[Lennart Borgman]

On Sat, Jun 12, 2010 at 11:05 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Lennart Borgman <lennart.borgman@gmail.com>
>> Date: Sat, 12 Jun 2010 22:07:36 +0200
>> Cc: emacs-devel@gnu.org
>>
>> >  abcde EDCBA xyz
>> >  ---------
>> >
>> > Now, what characters do you think should be displayed with the region
>> > face?
>>
>>
>> As I said in another message it would be "abcde CBA", i.e. exactly the
>> visual region that you now get when moving with the arrow keys so that
>> you get to C.
>
> But that doesn't make sense: the cursor traverses characters that you
> don't paint in the region face, and the region face is put on
> characters the cursor didn't move through.

So you are going to sit there painting them one by one just to be sure?

Re: arrow keys vs. C-f/b/n/p

[Lennart Borgman]

On Sat, Jun 12, 2010 at 11:08 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>
>>  And of course: without letting the arrow keys move visually on the
>> screen there is no easy way at all to move visually on the screen.
>
> I see nothing non-easy with logical-order motion.
>
>> This is actually a bit fun. Try creating with only blanks, just
>> spaces, tabs and newlines. Set some of the those chars to arabic and
>> let some be latin. Now try moving around the page using only the
>> visual keys. Try to go to the end of the page. (I suggest this as an
>> alternative to sudoku...)
>
> You can mock it as much as you want, but users of R2L scripts do this
> every day, and have no problem at all.


Of course, if you deny that there are no difficulties with having no
way to get visual movements then you can't see the problem.


> Anyway, we are beginning to repeat ourselves.  It's clear we disagree,
> so let's leave it at that.


I think a sum up is necessary. Do you want to make a sum up of the arguments?

Re: arrow keys vs. C-f/b/n/p

[Kenichi Handa]

This is not directly related to the current topic but...

In article <83ljakliqt.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> The current Emacs supports two text directions and two paragraph
> directions.  Both can be either left-to-right (L2R) or right-to-left
> (R2L).

> Paragraph direction determines whether the text starts at the left
> window margin or at the right margin.  Text direction determines
> the direction in which text is read.

How about using the word "orientation" for "paragraph
direction" to avoid confusion?  I my experiense, many people
misunderstood that even latin letters are written from right
to left in a paragraph (or a buffer) of "r2l direction".
After I started using the word "orientation", it seems that
such miunderstandings decreased.

---
Kenichi Handa
handa@m17n.org

RE: arrow keys vs. C-f/b/n/p

[Davis Herring]

>  They act the same as they always have: forward typically means
(continues to
> mean) _toward eob_, even in a bidi setting.  That's not obvious.

I think it is obvious, given the premise that Emacs is not going to treat
R2L text as a second-class citizen: why would "forward" be associated with
anything other than toward EOB?  (In particular, the only other
possibility seems to be for "forward" to be a synonym for "right".  That
would make Emacs' bidi support something of a farce, for reasons that
become pretty obvious once you consider things like `forward-paragraph'.)

Davis

-- 
This product is sold by volume, not by mass.  If it appears too dense or
too sparse, it is because mass-energy conversion has occurred during
shipping.

Re: arrow keys vs. C-f/b/n/p

[James Cloos]

>>>>> "AS" == Andreas Schwab <schwab@linux-m68k.org> writes:

AS> James Cloos <cloos@jhcloos.com> writes:
>> In gedit C-f and C-b move forward and backward logically

AS> By default, C-f and C-b in gedit don't do cursor motion.

True.  I have:

,----
| gtk-key-theme-name = "Emacs"
`----

in my .gtkrc-2.0, which enables C-f, C-b, C-n, C-f, C-a, C-e and perhaps
some more.

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6

Re: arrow keys vs. C-f/b/n/p

[Eli Zaretskii]

> From: Kenichi Handa <handa@m17n.org>
> Cc: emacs-devel@gnu.org
> Date: Sun, 13 Jun 2010 07:27:33 +0900
> 
> > Paragraph direction determines whether the text starts at the left
> > window margin or at the right margin.  Text direction determines
> > the direction in which text is read.
> 
> How about using the word "orientation" for "paragraph
> direction" to avoid confusion?

"Paragraph direction" is the term used by UAX#9.  I'm trying to use
terminology that is accepted there, so people who know about this or
want to read the description of the Unicode Bidirectional Algorithm
would not be confused.

> many people misunderstood that even latin letters are written from
> right to left in a paragraph (or a buffer) of "r2l direction".

??? Latin text is always displayed left to right, independent of the
paragraph direction.  More generally, text direction is independent of
the paragraph direction.

Re: arrow keys vs. C-f/b/n/p

[Lennart Borgman]

On Sat, Jun 12, 2010 at 11:16 PM, Lennart Borgman
<lennart.borgman@gmail.com> wrote:
> On Sat, Jun 12, 2010 at 11:05 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>>> From: Lennart Borgman <lennart.borgman@gmail.com>
>>> Date: Sat, 12 Jun 2010 22:07:36 +0200
>>> Cc: emacs-devel@gnu.org
>>>
>>> >  abcde EDCBA xyz
>>> >  ---------
>>> >
>>> > Now, what characters do you think should be displayed with the region
>>> > face?
>>>
>>>
>>> As I said in another message it would be "abcde CBA", i.e. exactly the
>>> visual region that you now get when moving with the arrow keys so that
>>> you get to C.
>>
>> But that doesn't make sense: the cursor traverses characters that you
>> don't paint in the region face, and the region face is put on
>> characters the cursor didn't move through.
>
> So you are going to sit there painting them one by one just to be sure?


I must admit that your point here makes more sense than I first thought.

I looked a bit at how things are setup in OpenOffice. The default
seems to be that arrow keys follows logical direction (and I could not
find how to change it, but it should be possible ... ;-).

They seem to have discussed this three years ago so I started
wondering why they left it this way. Clearly, it is not hard to see
the advantage of letting arrow keys move visually so what big win did
they see having them move logically.

I wonder if your point here, is the main advantage. The reason I
suspect this is that moving with the arrow keys shifted is the only
way to select text there. And during this operation you might more
often actually read the text.

So here actually (to my surprise) "reading is (quite often) cursoring".


Still, for other operations, I think that the reverse visual direction
for the arrow keys is bad.

Re: arrow keys vs. C-f/b/n/p

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Lennart Borgman <lennart.borgman@gmail.com>
>> Date: Sat, 12 Jun 2010 22:07:36 +0200
>> Cc: emacs-devel@gnu.org
>> 
>> >  abcde EDCBA xyz
>> >  ---------
>> >
>> > Now, what characters do you think should be displayed with the region
>> > face?
>> 
>> 
>> As I said in another message it would be "abcde CBA", i.e. exactly the
>> visual region that you now get when moving with the arrow keys so that
>> you get to C.
>
> But that doesn't make sense: the cursor traverses characters that you
> don't paint in the region face, and the region face is put on
> characters the cursor didn't move through.

I think "that doesn't make sense" will be unavoidable for some narrow
meaning of "sense" for some combinations of keypresses in mixed L2R-R2L
settings.

For me, the behavior makes perfect sense, and is, in some manner, even
telling us something beautiful about the structure of texts, just like
the equations of Special Relativity as opposed to the equations of
Newtonian mechanics "don't make sense" in some view, but tell us
something beautiful about the structure of spacetime.

We really should not try to define "makes sense" in terms of relations
that arise from fundamental properties of unidirectional text.

-- 
David Kastrup

Re: arrow keys vs. C-f/b/n/p

[Kenichi Handa]

In article <838w6jk52k.fsf@gnu.org>, Eli Zaretskii <eliz@gnu.org> writes:

> "Paragraph direction" is the term used by UAX#9.  I'm trying to use
> terminology that is accepted there, so people who know about this or
> want to read the description of the Unicode Bidirectional Algorithm
> would not be confused.

Hmmm, ok, it's up to you.

> > many people misunderstood that even latin letters are written from
> > right to left in a paragraph (or a buffer) of "r2l direction".

> ??? Latin text is always displayed left to right, independent of the
> paragraph direction.  More generally, text direction is independent of
> the paragraph direction.

Of course I know that.  But, I remember I got several
questions why they are so.

---
Kenichi Handa
handa@m17n.org

Re: arrow keys vs. C-f/b/n/p

[Lennart Borgman]

On Sun, Jun 13, 2010 at 12:24 PM, Kenichi Handa <handa@m17n.org> wrote:
>
>> > many people misunderstood that even latin letters are written from
>> > right to left in a paragraph (or a buffer) of "r2l direction".
>
>> ??? Latin text is always displayed left to right, independent of the
>> paragraph direction.  More generally, text direction is independent of
>> the paragraph direction.
>
> Of course I know that.  But, I remember I got several
> questions why they are so.


I do not understand what you mean with "even latin letters are written
from right to left". Could you please explain, perhaps with a small
example?

Re: arrow keys vs. C-f/b/n/p

[Eli Zaretskii]

> From: Lennart Borgman <lennart.borgman@gmail.com>
> Date: Sun, 13 Jun 2010 17:28:37 +0200
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> 
> On Sun, Jun 13, 2010 at 12:24 PM, Kenichi Handa <handa@m17n.org> wrote:
> >
> >> > many people misunderstood that even latin letters are written from
> >> > right to left in a paragraph (or a buffer) of "r2l direction".
> >
> >> ??? Latin text is always displayed left to right, independent of the
> >> paragraph direction.  More generally, text direction is independent of
> >> the paragraph direction.
> >
> > Of course I know that.  But, I remember I got several
> > questions why they are so.
> 
> 
> I do not understand what you mean with "even latin letters are written
> from right to left".

I guess Handa-san means to say that some people somehow got confused
to think that Latin text is displayed right to left in a R2L
paragraph, because they thought (incorrectly) that paragraph direction
overrides the normal text direction.

Re: arrow keys vs. C-f/b/n/p

[Lennart Borgman]

On Sun, Jun 13, 2010 at 7:20 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>
> I guess Handa-san means to say that some people somehow got confused
> to think that Latin text is displayed right to left in a R2L
> paragraph, because they thought (incorrectly) that paragraph direction
> overrides the normal text direction.

Thanks,  yes, that makes sense.

Re: arrow keys vs. C-f/b/n/p

[Eli Zaretskii]

> From: Chong Yidong <cyd@stupidchicken.com>
> Cc: Eli Zaretskii <eliz@gnu.org>, David Kastrup <dak@gnu.org>,
>         emacs-devel@gnu.org
> Date: Fri, 11 Jun 2010 15:50:39 -0400
> 
> Also, I would like to know what existing bidi editors do for
> Ctrl-left/right.

In gedit, left and right arrows work like they do now in Emacs,
i.e. they are sensitive to the paragraph direction.  However, C-left
and C-right are not sensitive to the paragraph direction.  IOW, in a
R2L paragraph, <left> moves to the left, but C-<left> moves by words
_to_the_right_!  So much for consistency (in Pango, I guess).

Re: Current head won't build

[Juri Linkov]

> Maybe "make bootstrap" should do some sort of "nearly-distclean"
> that doesn't wipe out the configuration file I just generated?

It seems "make bootstrap" doesn't delete the configuration file
and currently compiles successfully.

-- 
Juri Linkov
http://www.jurta.org/emacs/

Re: Current head won't build

[Randal L. Schwartz]

>>>>> "Juri" == Juri Linkov <juri@jurta.org> writes:

>> Maybe "make bootstrap" should do some sort of "nearly-distclean"
>> that doesn't wipe out the configuration file I just generated?

Juri> It seems "make bootstrap" doesn't delete the configuration file
Juri> and currently compiles successfully.

I think you misunderstood me.  Apparently, "make bootstrap" doesn't
clean out *enough*.  When I finally got rid of everything and wound back
to just the distribution, and configured forward from there, it worked
fine. So something was left around from a previous build that triggered
my initial complaint.

-- 
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Smalltalk/Perl/Unix consulting, Technical writing, Comedy, etc. etc.
See http://methodsandmessages.vox.com/ for Smalltalk and Seaside discussion

Re: Current head won't build

[Andreas Schwab]

merlyn@stonehenge.com (Randal L. Schwartz) writes:

> I think you misunderstood me.  Apparently, "make bootstrap" doesn't
> clean out *enough*.

Are you sure your sources were up-to-date?

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Current head won't build

[Randal L. Schwartz]

>>>>> "Andreas" == Andreas Schwab <schwab@linux-m68k.org> writes:

Andreas> merlyn@stonehenge.com (Randal L. Schwartz) writes:
>> I think you misunderstood me.  Apparently, "make bootstrap" doesn't
>> clean out *enough*.

Andreas> Are you sure your sources were up-to-date?

Absolutely.  I trust "git checkout SOMECOMMIT" to do the right thing.

-- 
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Smalltalk/Perl/Unix consulting, Technical writing, Comedy, etc. etc.
See http://methodsandmessages.vox.com/ for Smalltalk and Seaside discussion

Re: Current head won't build

[Andreas Schwab]

merlyn@stonehenge.com (Randal L. Schwartz) writes:

>>>>>> "Andreas" == Andreas Schwab <schwab@linux-m68k.org> writes:
>
> Andreas> merlyn@stonehenge.com (Randal L. Schwartz) writes:
>>> I think you misunderstood me.  Apparently, "make bootstrap" doesn't
>>> clean out *enough*.
>
> Andreas> Are you sure your sources were up-to-date?
>
> Absolutely.  I trust "git checkout SOMECOMMIT" to do the right thing.

That does not necessarily mean that you were up-to-date.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Current head won't build

[Teemu Likonen]

* 2010-06-15 21:25 (+0200), Andreas Schwab wrote:

> merlyn@stonehenge.com (Randal L. Schwartz) writes:
>> Absolutely. I trust "git checkout SOMECOMMIT" to do the right thing.
>
> That does not necessarily mean that you were up-to-date.

Yes, the Emacs Git mirror may not be up-to-date. Also, sometimes
problems may be caused by dirty working directory. In a Git repository
one gets really clean state with commands

    git reset --hard HEAD       # or SOMECOMMIT
    git clean -fdx

(Don't execute those commands unless you know what they do!)

But I have no idea what is causing the problem here.

Re: Current head won't build

[Andreas Schwab]

Teemu Likonen <tlikonen@iki.fi> writes:

> * 2010-06-15 21:25 (+0200), Andreas Schwab wrote:
>
>> merlyn@stonehenge.com (Randal L. Schwartz) writes:
>>> Absolutely. I trust "git checkout SOMECOMMIT" to do the right thing.
>>
>> That does not necessarily mean that you were up-to-date.
>
> Yes, the Emacs Git mirror may not be up-to-date.

I mean up-to-date wrt. current development.  Especially 53c4f4cc8c
should be present.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Current head won't build

[Randal L. Schwartz]

>>>>> "Teemu" == Teemu Likonen <tlikonen@iki.fi> writes:

Teemu> Yes, the Emacs Git mirror may not be up-to-date. Also, sometimes
Teemu> problems may be caused by dirty working directory.

Except that I had been moving *forward*.  If "make clean bootstrap" does
not do the right thing when moving forward, it's really broken.  I'd
also argue it should do the right thing even when moving backward, but I
can see that might not be possible.

-- 
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Smalltalk/Perl/Unix consulting, Technical writing, Comedy, etc. etc.
See http://methodsandmessages.vox.com/ for Smalltalk and Seaside discussion

Re: Current head won't build

[Stephen J. Turnbull]

Randal L. Schwartz writes:

 > Except that I had been moving *forward*.  If "make clean bootstrap" does
 > not do the right thing when moving forward, it's really broken.

"make clean bootstrap" is broken, asking for trouble.  "make clean" by
design does not get you to a state suitable for bootstrapping:

## make clean  or  make mostlyclean
##      Delete all files from the current directory that are normally
##      created by building the program.  Don't delete the files that
##      record the configuration.  Also preserve files that could be made
##      by building, but normally aren't because the distribution comes
##      with them.

##      Delete `.dvi' files here if they are not part of the distribution.

## make distclean
##      Delete all files from the current directory that are created by
##      configuring or building the program.  If you have unpacked the
##      source and built the program without creating any other files,
##      `make distclean' should leave only the files that were in the
##      distribution.

You want at least "make distclean bootstrap".

Re: Current head won't build

[Andreas Schwab]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> You want at least "make distclean bootstrap".

distclean will nuke your makefiles and configuration.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Current head won't build

[Juanma Barranquero]

On Wed, Jun 16, 2010 at 10:21, Andreas Schwab <schwab@linux-m68k.org> wrote:

> distclean will nuke your makefiles and configuration.

Yes, but it is the safest route. Currently, to bootstrap I do

  bzr clean-tree --unknown --ignored --detritus --force
  configure ;; with suitable args
  make bootstrap info install

which is slower, but I get to check that fresh installs are bootstrappable.

    Juanma

Re: Current head won't build

[Stephen J. Turnbull]

Andreas Schwab writes:

 > "Stephen J. Turnbull" <stephen@xemacs.org> writes:
 > 
 > > You want at least "make distclean bootstrap".
 > 
 > distclean will nuke your makefiles and configuration.

Yup.  That's precisely what is desired here.  Scorched earth, build
the world from scratch.  Otherwise, you *will* occasionally lose.

Most people are happy losing very occasionally if it saves a fair
amount of time on each pseudo-bootstrap, but AIUI Randal professes a
far more exacting standard.  Either that, or he's pretty unlucky and
loses on a fairly large fraction of his bootstraps.

Anyway, "make distclean" is the way to minimize bootstrap lossage.

As far as the ./configure line itself goes, sure, you want to save
that.  But there are plenty of ways to do that.

Re: Current head won't build

[Andreas Schwab]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> Andreas Schwab writes:
>
>  > "Stephen J. Turnbull" <stephen@xemacs.org> writes:
>  > 
>  > > You want at least "make distclean bootstrap".
>  > 
>  > distclean will nuke your makefiles and configuration.
>
> Yup.  That's precisely what is desired here.

What is bootstrap supposed to do with the unconfigured tree?

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Current head won't build

[Andreas Schwab]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> Anyway, "make distclean" is the way to minimize bootstrap lossage.

bootstrap-clean does already di everything needed to start a bootstrap
without nuking your configuration.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Current head won't build

[Randal L. Schwartz]

>>>>> "Andreas" == Andreas Schwab <schwab@linux-m68k.org> writes:

Andreas> bootstrap-clean does already di everything needed to start a bootstrap
Andreas> without nuking your configuration.

But would it have also done a nearly distclean?   As in, would
it have fixed the problem that started this thread?

-- 
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Smalltalk/Perl/Unix consulting, Technical writing, Comedy, etc. etc.
See http://methodsandmessages.vox.com/ for Smalltalk and Seaside discussion

Re: Current head won't build

[Andreas Schwab]

merlyn@stonehenge.com (Randal L. Schwartz) writes:

> But would it have also done a nearly distclean?

Sure.

> As in, would it have fixed the problem that started this thread?

I'm still waiting for an answer to the question whether the source tree
was up-to-date wrt. 53c4f4cc.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Current head won't build

[Randal L. Schwartz]

>>>>> "Andreas" == Andreas Schwab <schwab@linux-m68k.org> writes:

Andreas> I'm still waiting for an answer to the question whether the source tree
Andreas> was up-to-date wrt. 53c4f4cc.

I was bisecting back and forth from the previous day to the current day.
The current day failed, the previous day worked, and after 5 bisects,
found the build that failed first.

I was doing "make clean bootstrap", but not "distclean". Thus, something
in that build and later fails on a tree that contains prior work
products.

-- 
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Smalltalk/Perl/Unix consulting, Technical writing, Comedy, etc. etc.
See http://methodsandmessages.vox.com/ for Smalltalk and Seaside discussion

Re: Current head won't build

[Randal L. Schwartz]

>>>>> "Andreas" == Andreas Schwab <schwab@linux-m68k.org> writes:

Andreas> bootstrap-clean does already di everything needed to start a bootstrap
Andreas> without nuking your configuration.

There's nothing in INSTALL or INSTALL.BZR about bootstrap-clean.
Time to update the docs?

-- 
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Smalltalk/Perl/Unix consulting, Technical writing, Comedy, etc. etc.
See http://methodsandmessages.vox.com/ for Smalltalk and Seaside discussion

Re: Current head won't build

[Andreas Schwab]

Since you are unable to answer my question I am unable to answer yours.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Current head won't build

[Andreas Schwab]

merlyn@stonehenge.com (Randal L. Schwartz) writes:

> There's nothing in INSTALL or INSTALL.BZR about bootstrap-clean.

Neither are a lot of other internal makefile targets.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Current head won't build

[Randal L. Schwartz]

>>>>> "Andreas" == Andreas Schwab <schwab@linux-m68k.org> writes:

Andreas> Since you are unable to answer my question I am unable to
Andreas> answer yours

I believe I was forthright in my answer.  A few days ago, I did a git
checkout of the current HEAD.  It failed to build, even with "make clean
bootstrap".  So I bisected back to the previous day's good build to find
the culprit, and posted that one here.

I then did a "distclean", and even the broken one built OK.  So
therefore, there's something that was introduced recently that required
a distclean when built in a tree containing an older version.

Other than playing back a screen log, how much clearer can I be?

Now, will you please update the INSTALL.BZR doc to mention
"bootstrap-clean", so that nobody else needs to repeat these steps?

-- 
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Smalltalk/Perl/Unix consulting, Technical writing, Comedy, etc. etc.
See http://methodsandmessages.vox.com/ for Smalltalk and Seaside discussion

Re: Current head won't build

[Andreas Schwab]

merlyn@stonehenge.com (Randal L. Schwartz) writes:

> Now, will you please update the INSTALL.BZR doc to mention
> "bootstrap-clean", so that nobody else needs to repeat these steps?

It's already there.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Current head won't build

[Randal L. Schwartz]

>>>>> "Andreas" == Andreas Schwab <schwab@linux-m68k.org> writes:

Andreas> merlyn@stonehenge.com (Randal L. Schwartz) writes:
>> Now, will you please update the INSTALL.BZR doc to mention
>> "bootstrap-clean", so that nobody else needs to repeat these steps?

Andreas> It's already there.

No it isn't.

    Locohost.local:~/MIRROR/emacs-GIT % grep bootstrap INSTALL.BZR 
    before bootstrapping:
    build from Bazaar you must run "make bootstrap" instead of just "make":
      $ make bootstrap
    Normally, it is not necessary to use "make bootstrap" after every
    If either of the above partial procedures fails, try "make bootstrap".
    etc.) before "make bootstrap" or "make"; the rest of the procedure is
    Locohost.local:~/MIRROR/emacs-GIT % git log -1
    commit ef7d88db17e9c2acbe8c3b71d6b13aa73bb133ba
    Author: Stefan Monnier <monnier@iro.umontreal.ca>
    Date:   Wed Jun 16 10:10:02 2010 -0400

        * src/lread.c (read1): Phase out old-style backquotes a bit more.

If it *had* been there, I would have incorporated it into my build
process long ago.

-- 
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Smalltalk/Perl/Unix consulting, Technical writing, Comedy, etc. etc.
See http://methodsandmessages.vox.com/ for Smalltalk and Seaside discussion

Re: Current head won't build

[Andreas Schwab]

merlyn@stonehenge.com (Randal L. Schwartz) writes:

> If it *had* been there, I would have incorporated it into my build
> process long ago.

You've used it all the time already.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Current head won't build

[Randal L. Schwartz]

>>>>> "Andreas" == Andreas Schwab <schwab@linux-m68k.org> writes:

Andreas> merlyn@stonehenge.com (Randal L. Schwartz) writes:
>> If it *had* been there, I would have incorporated it into my build
>> process long ago.

Andreas> You've used it all the time already.

If it's part of "make clean" or "make bootstrap", then it's
insufficient.  Thus, my initial bug report.

-- 
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Smalltalk/Perl/Unix consulting, Technical writing, Comedy, etc. etc.
See http://methodsandmessages.vox.com/ for Smalltalk and Seaside discussion

Re: Current head won't build

[Andreas Schwab]

merlyn@stonehenge.com (Randal L. Schwartz) writes:

> If it's part of "make clean" or "make bootstrap", then it's
> insufficient.  Thus, my initial bug report.

Have you tried with 53c4f4cc?

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Current head won't build

[Stefan Monnier]

Andreas> bootstrap-clean does already di everything needed to start a bootstrap
Andreas> without nuking your configuration.

> There's nothing in INSTALL or INSTALL.BZR about bootstrap-clean.
> Time to update the docs?

"make bootstrap" should work.  If it doesn't, it's a bug.
You should never need to run "make bootstrap-clean", for example, since
it's done by "make bootstrap" for you.


        Stefan

Re: Current head won't build

[Randal L. Schwartz]

>>>>> "Stefan" == Stefan Monnier <monnier@iro.umontreal.ca> writes:

Stefan> "make bootstrap" should work.  If it doesn't, it's a bug.
Stefan> You should never need to run "make bootstrap-clean", for example, since
Stefan> it's done by "make bootstrap" for you.

Yup.

I haven't had time to rebuild last week's dir and try the latest release
over it... hopefully I can get to that later today to ensure the current
"make bootstrap" does the job.

-- 
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Smalltalk/Perl/Unix consulting, Technical writing, Comedy, etc. etc.
See http://methodsandmessages.vox.com/ for Smalltalk and Seaside discussion

Re: Current head won't build

[Stephen J. Turnbull]

Andreas Schwab writes:
 > "Stephen J. Turnbull" <stephen@xemacs.org> writes:
 > 
 > > Anyway, "make distclean" is the way to minimize bootstrap lossage.
 > 
 > bootstrap-clean does already di everything needed to start a bootstrap
 > without nuking your configuration.

Nuking the configuration may be necessary (eg, if config.h.in changes).

Even if *that* triggers reconfiguration automatically, I find it hard
to believe that Emacs's Makefiles are sufficiently finicky that all
potentially relevant changes trigger reconfiguration.

I am not suggesting that "make bootstrap" should be changed, just that
if it doesn't work, "make clean bootstrap" is not going to help.

Re: Current head won't build

[Stephen J. Turnbull]

Andreas Schwab writes:

 > What is bootstrap supposed to do with the unconfigured tree?

Tell the user to run ./configure.

Re: Current head won't build

[Andreas Schwab]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> Andreas Schwab writes:
>
>  > What is bootstrap supposed to do with the unconfigured tree?
>
> Tell the user to run ./configure.

IOW, your "solution" is guaranteed to fail.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Current head won't build

[Stephen J. Turnbull]

Andreas Schwab writes:
 > "Stephen J. Turnbull" <stephen@xemacs.org> writes:
 > 
 > > Andreas Schwab writes:
 > >
 > >  > What is bootstrap supposed to do with the unconfigured tree?
 > >
 > > Tell the user to run ./configure.
 > 
 > IOW, your "solution" is guaranteed to fail.

No, actually, it works fine for me.  I've always done re-bootstraps
that way (admittedly, I've never tried a naked "make bootstrap" in
Emacs, I just use a script adapted from my XEmacs habits, which has
never had bootstrap target, although make beta comes close nowadays).
My customary configure options are embedded in the script, which does

# code to assemble configure options in OPTIONS

ln -sf beta.err.`date +%Y%m%d%H%M%S` beta.err
if test -e Makefile; then make distclean; fi
../configure >beta.err 2>&1 $OPTIONS \
&& make bootstrap >>beta.err 2>&1

Except that the user might not have saved off his configure line, I
really don't understand what the objection to distclean is.  It's not
like the configure step takes 10 times as much time as "make bootstrap"
itself does!

Re: Current head won't build

[Andreas Schwab]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> No, actually, it works fine for me.

How can it "work" if make distclean bootstrap fails?

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Current head won't build

[Stephen J. Turnbull]

Andreas Schwab writes:
 > "Stephen J. Turnbull" <stephen@xemacs.org> writes:
 > 
 > > No, actually, it works fine for me.
 > 
 > How can it "work" if make distclean bootstrap fails?

It can't.  But the real point is that "make clean" is definitely no
help.  And I think it's quite possible that make distclean, followed
by configuring, sometimes will lead to a success where a simple "make
bootstrap" in a previous build would fail.

Re: Current head won't build

[Andreas Schwab]

"Stephen J. Turnbull" <stephen@xemacs.org> writes:

> And I think it's quite possible that make distclean, followed by
> configuring, sometimes will lead to a success where a simple "make
> bootstrap" in a previous build would fail.

Unlikely in my experience.  Everytime I saw bootstrap fail so far it was
a genuine bug that won't be fixed by using a pristine tree.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."