Mixed L2R and R2L paragraphs and horizontal scroll

Mixed L2R and R2L paragraphs and horizontal scroll

[Eli Zaretskii]

Here's a peculiar design decision that needs to be made for the bidi
display: how to display horizontally scrolled lines in a buffer with
mixed L2R and R2L paragraphs.

Let me start with an example of 2 single-line paragraphs (the frame
around the text shows the window borders):

      +---------------------------------------+
      |abcdefghijk                            |
      |                            KJIHGFEDCBA|
      |                                       |
      +---------------------------------------+

Looks okay, right?  But what if I split the window horizontally (with
"C-x 3"):

      +-------------------+-------------------+
      |abcdefghijk        |abcdefghijk        |
      |        KJIHGFEDCBA|        KJIHGFEDCBA|
      |                   |                   |
      +-------------------+-------------------+

Does this look correct?  Is anyone bothered by the fact that the
relative horizontal position of the two lines has changed, and yet
there are no line truncation glyphs anywhere in sight?

Some word processors have a rigid length of a line: they display a
ruler somewhere above or below the text, and therefore a line's length
is known as some fixed multiple of a standard character size.  In
these word processors, changing the window width preserves the
relative horizontal position of L2R and R2L lines.  But Emacs does not
have a rigid line length, so flushing the R2L lines against the right
margin of the window makes their position depend on the window width.
Is that okay?  (I think it is.)

Now let's split the leftmost window again, so lines get truncated:

      +---------+---------+-------------------+
      |abcdefgh$|abcdefgh$|abcdefghijk        |
      |$HGFEDCBA|$HGFEDCBA|        KJIHGFEDCBA|
      |         |         |                   |
      +---------+---------+-------------------+

Does this still look okay?  (I think it does.)

Now the crucial question: what do you think should happen if I scroll
the leftmost window horizontally so that the end of the first line
(the characters "ijk") become visible?  I think it should be this:

      +---------+---------+-------------------+
      |$ijk     |abcdefgh$|abcdefghijk        |
      |$HGFEDCBA|$HGFEDCBA|        KJIHGFEDCBA|
      |         |         |                   |
      +---------+---------+-------------------+

Note that the display of the second line in the leftmost window did
not change, and as result, the amounts of horizontal scroll in the 1st
and the 2nd lines are different.  This is unlike in the current
unidirectional display, where all the lines are always hscrolled by
the same amount.

Does this behavior look reasonable?  If not, what are the
alternatives?  Note that any alternative should be consistent about
the fact that Emacs does not have a rigid line length, therefore any
initial relative horizontal position of characters (before we split
the window) is arbitrary, and is a function of the window's initial
arbitrary width.

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[David De La Harpe Golden]

Eli Zaretskii wrote:

> Does this behavior look reasonable?  If not, what are the
> alternatives?  Note that any alternative should be consistent about
> the fact that Emacs does not have a rigid line length, therefore any
> initial relative horizontal position of characters (before we split
> the window) is arbitrary, and is a function of the window's initial
> arbitrary width.
> 
> 


Eh, not saying it's a good alternative, maybe one could just _introduce_ 
a hard R2L paragraph start column setting, not  coupled to window width 
/or L2R line length/, any more than fill-column being 70 stops 
particular L2R lines being longer than 70. In some cases, unwrapped long 
L2R lines might thus spill past the R2L start column.
...Probably pretty obnoxious...

setting R2L start at max(render length of L2R lines given adequate 
space, or window width iff wrapping on)+C (C probably at or near 0) is 
not quite the same thing as a word-processor page-dimensions-based rigid 
length and could maybe work if that was known for the whole buffer, but 
as per other recent thread, emacs in particular probably doesn't have 
the luxury of such info in general.

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[Eli Zaretskii]

> Date: Sat, 30 Jan 2010 15:14:28 +0000
> From: David De La Harpe Golden <david@harpegolden.net>
> CC: emacs-devel@gnu.org, emacs-bidi@gnu.org
> 
> Eh, not saying it's a good alternative, maybe one could just _introduce_ 
> a hard R2L paragraph start column setting, not  coupled to window width 
> /or L2R line length/, any more than fill-column being 70 stops 
> particular L2R lines being longer than 70. In some cases, unwrapped long 
> L2R lines might thus spill past the R2L start column.

Yes, and in other cases, the R2L start column will be less than the
window width, making the R2L lines hang in the middle of a line,
instead of being flushed all the way to the right.

I'm not sure this is better.

Thanks for the feedback.

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[martin rudalics]

 > Note that the display of the second line in the leftmost window did
 > not change, and as result, the amounts of horizontal scroll in the 1st
 > and the 2nd lines are different.  This is unlike in the current
 > unidirectional display, where all the lines are always hscrolled by
 > the same amount.

What should `window-hscroll' return in that case?  The value of the 1st
line, the value of the line where `point' is on, ...

martin

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[Eli Zaretskii]

> Date: Sat, 30 Jan 2010 17:36:04 +0100
> From: martin rudalics <rudalics@gmx.at>
> CC: emacs-devel@gnu.org, emacs-bidi@gnu.org
> 
>  > Note that the display of the second line in the leftmost window did
>  > not change, and as result, the amounts of horizontal scroll in the 1st
>  > and the 2nd lines are different.  This is unlike in the current
>  > unidirectional display, where all the lines are always hscrolled by
>  > the same amount.
> 
> What should `window-hscroll' return in that case?

Why should it change from what it returns now?  Its doc string says:

   Return the number of columns by which WINDOW is scrolled from left margin.
   WINDOW defaults to the selected window.

So if the window is scrolled from the left margin, either because
point is on a L2R line on a column that's beyond the window's right
margin, or because the user scrolled with "C-x <", it will return the
amount of scroll as it does today.  If the window is scrolled from the
right margin, it could return a negative value, I suppose.  Or we
could have a separate function for that.  It really depends on what
the users of window-hscroll do with the value.

Or are you saying that we should scroll the R2L lines to the right by
as much as we scroll the L2R lines to the left?  That is, are you
saying we should display the last example as


      +---------+---------+-------------------+
      |$ijk     |abcdefgh$|abcdefghijk        |
      |     KJI$|$HGFEDCBA|        KJIHGFEDCBA|
      |         |         |                   |
      +---------+---------+-------------------+

Is this what you are suggesting?

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[martin rudalics]

 > Why should it change from what it returns now?

Because it's no more a window but a line property, IIUC.  When, after
scrolling the first line, you move `point' to show another portion of
the second line like this

       +---------+---------+-------------------+
       |$ijk     |abcdefgh$|abcdefghijk        |
       |KJIHGFED$|$HGFEDCBA|        KJIHGFEDCBA|
       |         |         |                   |
       +---------+---------+-------------------+

you get two different non-zero values of `window-hscroll'.  Or am I
missing something?

 > Or are you saying that we should scroll the R2L lines to the right by
 > as much as we scroll the L2R lines to the left?  That is, are you
 > saying we should display the last example as
 >
 >
 >       +---------+---------+-------------------+
 >       |$ijk     |abcdefgh$|abcdefghijk        |
 >       |     KJI$|$HGFEDCBA|        KJIHGFEDCBA|
 >       |         |         |                   |
 >       +---------+---------+-------------------+
 >
 > Is this what you are suggesting?

This would be needlessly inconvenient.  Hence, `window-hscroll' should
probably return the value for the line `window-point' is on.

martin

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[Eli Zaretskii]

> Date: Sat, 30 Jan 2010 18:22:27 +0100
> From: martin rudalics <rudalics@gmx.at>
> CC: emacs-devel@gnu.org, emacs-bidi@gnu.org
> 
>  > Why should it change from what it returns now?
> 
> Because it's no more a window but a line property, IIUC.  When, after
> scrolling the first line, you move `point' to show another portion of
> the second line like this
> 
>        +---------+---------+-------------------+
>        |$ijk     |abcdefgh$|abcdefghijk        |
>        |KJIHGFED$|$HGFEDCBA|        KJIHGFEDCBA|
>        |         |         |                   |
>        +---------+---------+-------------------+
> 
> you get two different non-zero values of `window-hscroll'.

No, you don't necessarily get two different values.  The value from
the second line defines a scroll from the right margin, not from the
left.  (Which probably means it's not a good idea to have
window-hscroll return a negative value in that case.)

>  > Or are you saying that we should scroll the R2L lines to the right by
>  > as much as we scroll the L2R lines to the left?  That is, are you
>  > saying we should display the last example as
>  >
>  >
>  >       +---------+---------+-------------------+
>  >       |$ijk     |abcdefgh$|abcdefghijk        |
>  >       |     KJI$|$HGFEDCBA|        KJIHGFEDCBA|
>  >       |         |         |                   |
>  >       +---------+---------+-------------------+
>  >
>  > Is this what you are suggesting?
> 
> This would be needlessly inconvenient.

Why inconvenient?  That's what would happen if the second line was
displayed at the left margin, like this:

      +---------+---------+-------------------+
      |$ijk     |abcdefgh$|abcdefghijk        |
      |$IJK     |$HGFEDCBA|        KJIHGFEDCBA|
      |         |         |                   |
      +---------+---------+-------------------+

The advantage is that all the lines scroll in lockstep, albeit in two
different directions, and `window-hscroll' can still return a single
value whose meaning is well defined.

> Hence, `window-hscroll' should probably return the value for the
> line `window-point' is on.

That's also possible.  Are the users of this function normally
interested only in the amount of scroll of the current line?

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[martin rudalics]

 >> Because it's no more a window but a line property, IIUC.  When, after
 >> scrolling the first line, you move `point' to show another portion of
 >> the second line like this
 >>
 >>        +---------+---------+-------------------+
 >>        |$ijk     |abcdefgh$|abcdefghijk        |
 >>        |KJIHGFED$|$HGFEDCBA|        KJIHGFEDCBA|
 >>        |         |         |                   |
 >>        +---------+---------+-------------------+
 >>
 >> you get two different non-zero values of `window-hscroll'.
 >
 > No, you don't necessarily get two different values.  The value from
 > the second line defines a scroll from the right margin, not from the
 > left.  (Which probably means it's not a good idea to have
 > window-hscroll return a negative value in that case.)

When you hscroll the second line we must be able to return that somehow
- otherwise how would we know?  So either we have to invent a new
function like `window-RL-hscroll' or interpret the return value of
`window-hscroll' appropriately.  And this function must probably handle
lines containing both LR and RL text correctly.

 >>  >       +---------+---------+-------------------+
 >>  >       |$ijk     |abcdefgh$|abcdefghijk        |
 >>  >       |     KJI$|$HGFEDCBA|        KJIHGFEDCBA|
 >>  >       |         |         |                   |
 >>  >       +---------+---------+-------------------+
 >>  >
 >>  > Is this what you are suggesting?
 >>
 >> This would be needlessly inconvenient.
 >
 > Why inconvenient?

Inconvenient for the user.  Why scroll something out of view if there's
no urgent need to do so.  (Although a similar argument could be made for
pure LR text as well - why scroll the entire window when I want to see
the last characters of the single overlong line in my entire text.  With
automatic line-wrapping turned off, I mean.)

 > That's what would happen if the second line was
 > displayed at the left margin, like this:
 >
 >       +---------+---------+-------------------+
 >       |$ijk     |abcdefgh$|abcdefghijk        |
 >       |$IJK     |$HGFEDCBA|        KJIHGFEDCBA|
          ^^^^^^^^^
 >       |         |         |                   |
 >       +---------+---------+-------------------+

I fail to understand the marked text in your example.

 > The advantage is that all the lines scroll in lockstep, albeit in two
 > different directions, and `window-hscroll' can still return a single
 > value whose meaning is well defined.

So it would be convenient for the designer.

 >> Hence, `window-hscroll' should probably return the value for the
 >> line `window-point' is on.
 >
 > That's also possible.  Are the users of this function normally
 > interested only in the amount of scroll of the current line?

Mostly for calculating the argument of `set-window-hscroll' I suppose.
But I have never looked into this.

martin

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[Eli Zaretskii]

>  >>  >       +---------+---------+-------------------+
>  >>  >       |$ijk     |abcdefgh$|abcdefghijk        |
>  >>  >       |     KJI$|$HGFEDCBA|        KJIHGFEDCBA|
>  >>  >       |         |         |                   |
>  >>  >       +---------+---------+-------------------+
>  >>  >
>  >>  > Is this what you are suggesting?
>  >>
>  >> This would be needlessly inconvenient.
>  >
>  > Why inconvenient?
> 
> Inconvenient for the user.  Why scroll something out of view if there's
> no urgent need to do so.  (Although a similar argument could be made for
> pure LR text as well

Exactly!  So perhaps we should adopt this design, as it doesn't make
things worse and does not require additional interfaces (such as
`window-RL-hscroll').

>  > That's what would happen if the second line was
>  > displayed at the left margin, like this:
>  >
>  >       +---------+---------+-------------------+
>  >       |$ijk     |abcdefgh$|abcdefghijk        |
>  >       |$IJK     |$HGFEDCBA|        KJIHGFEDCBA|
>           ^^^^^^^^^
>  >       |         |         |                   |
>  >       +---------+---------+-------------------+
> 
> I fail to understand the marked text in your example.

In the current unidirectional display, where text is always displayed
in its logical (i.e. reading) order, the second line would have been
displayed as "ABCDEFGHIJK", flushed to the left margin.  Then when the
line above it is scrolled to show just "$ijk", so would be the second
line, and it would display as "$IJK".

My point was that the bidirectional display conceptually does the
same, except it reverses the characters and flushes them to the right.

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[martin rudalics]

 >> Inconvenient for the user.  Why scroll something out of view if there's
 >> no urgent need to do so.  (Although a similar argument could be made for
 >> pure LR text as well
 >
 > Exactly!  So perhaps we should adopt this design,

Which one?

 > as it doesn't make
 > things worse and does not require additional interfaces (such as
 > `window-RL-hscroll').
 >
 >>  > That's what would happen if the second line was
 >>  > displayed at the left margin, like this:
 >>  >
 >>  >       +---------+---------+-------------------+
 >>  >       |$ijk     |abcdefgh$|abcdefghijk        |
 >>  >       |$IJK     |$HGFEDCBA|        KJIHGFEDCBA|
 >>           ^^^^^^^^^
 >>  >       |         |         |                   |
 >>  >       +---------+---------+-------------------+
 >>
 >> I fail to understand the marked text in your example.
 >
 > In the current unidirectional display, where text is always displayed
 > in its logical (i.e. reading) order, the second line would have been
 > displayed as "ABCDEFGHIJK", flushed to the left margin.  Then when the
 > line above it is scrolled to show just "$ijk", so would be the second
 > line, and it would display as "$IJK".

I still don't get you.  In your earlier examples capitalized text always
showed reverse as KJI and not as IJK.  But I also fail to understand why
there's no text on the right of this.  So maybe you meant something like

       +---------+---------+-------------------+
       |$ijk     |abcdefgh$|abcdefghijk        |
       | KJIHGFE$|        $|        KJIHGFEDCBA|
       |         |         |                   |
       +---------+---------+-------------------+

(no text in the second line of the second window) instead?

 > My point was that the bidirectional display conceptually does the
 > same, except it reverses the characters and flushes them to the right.

I suppose this would apply regardless of the strategy chosen.

martin

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[Eli Zaretskii]

> Date: Sat, 30 Jan 2010 20:45:50 +0100
> From: martin rudalics <rudalics@gmx.at>
> Cc: emacs-bidi@gnu.org, emacs-devel@gnu.org
> 
>  >> Inconvenient for the user.  Why scroll something out of view if there's
>  >> no urgent need to do so.  (Although a similar argument could be made for
>  >> pure LR text as well
>  >
>  > Exactly!  So perhaps we should adopt this design,
> 
> Which one?

The one that scrolls the R2L lines to the right when the L2R lines are
scrolled to the left.

>  >>  > That's what would happen if the second line was
>  >>  > displayed at the left margin, like this:
>  >>  >
>  >>  >       +---------+---------+-------------------+
>  >>  >       |$ijk     |abcdefgh$|abcdefghijk        |
>  >>  >       |$IJK     |$HGFEDCBA|        KJIHGFEDCBA|
>  >>           ^^^^^^^^^
>  >>  >       |         |         |                   |
>  >>  >       +---------+---------+-------------------+
>  >>
>  >> I fail to understand the marked text in your example.
>  >
>  > In the current unidirectional display, where text is always displayed
>  > in its logical (i.e. reading) order, the second line would have been
>  > displayed as "ABCDEFGHIJK", flushed to the left margin.  Then when the
>  > line above it is scrolled to show just "$ijk", so would be the second
>  > line, and it would display as "$IJK".
> 
> I still don't get you.  In your earlier examples capitalized text always
> showed reverse as KJI and not as IJK.

Yes, and they were also flushed to the right.  This is what the
bidirectional display does.  In the above example, the leftmost window
is displayed as with the current unidirectional display engine we have
in Emacs 23.  That is why it is flushed to the left and the R2L text
is not reversed.

> But I also fail to understand why
> there's no text on the right of this.  So maybe you meant something like
> 
>        +---------+---------+-------------------+
>        |$ijk     |abcdefgh$|abcdefghijk        |
>        | KJIHGFE$|        $|        KJIHGFEDCBA|
>        |         |         |                   |
>        +---------+---------+-------------------+
> 
> (no text in the second line of the second window) instead?

No, I meant what I wrote.  The middle window was not scrolled, and
there's no need to change anything in it just because the leftmost
window was scrolled.

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[martin rudalics]

 > Yes, and they were also flushed to the right.  This is what the
 > bidirectional display does.

With bidirectional display going to the end of the first line in the
first window you earlier proposed to show

       +---------+---------+-------------------+
       |$ijk     |abcdefgh$|abcdefghijk        |
       |$HGFEDCBA|$HGFEDCBA|        KJIHGFEDCBA|
       |         |         |                   |
       +---------+---------+-------------------+

Now moving in the first window to the end of the second line do you want
to display

       +---------+---------+-------------------+
       |$ijk     |abcdefgh$|abcdefghijk        |
       |     KJI$|$HGFEDCBA|        KJIHGFEDCBA|
       |         |         |                   |
       +---------+---------+-------------------+

or

       +---------+---------+-------------------+
       |abcdefgh$|abcdefgh$|abcdefghijk        |
       |     KJI$|$HGFEDCBA|        KJIHGFEDCBA|
       |         |         |                   |
       +---------+---------+-------------------+

or something else?

martin

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[Eli Zaretskii]

> Date: Sun, 31 Jan 2010 10:36:36 +0100
> From: martin rudalics <rudalics@gmx.at>
> CC: emacs-bidi@gnu.org, emacs-devel@gnu.org
> 
>  > Yes, and they were also flushed to the right.  This is what the
>  > bidirectional display does.
> 
> With bidirectional display going to the end of the first line in the
> first window you earlier proposed to show
> 
>        +---------+---------+-------------------+
>        |$ijk     |abcdefgh$|abcdefghijk        |
>        |$HGFEDCBA|$HGFEDCBA|        KJIHGFEDCBA|
>        |         |         |                   |
>        +---------+---------+-------------------+

Yes, I did.  But now I changed my mind, see below.

> Now moving in the first window to the end of the second line do you want
> to display
> 
>        +---------+---------+-------------------+
>        |$ijk     |abcdefgh$|abcdefghijk        |
>        |     KJI$|$HGFEDCBA|        KJIHGFEDCBA|
>        |         |         |                   |
>        +---------+---------+-------------------+
> 
> or
> 
>        +---------+---------+-------------------+
>        |abcdefgh$|abcdefgh$|abcdefghijk        |
>        |     KJI$|$HGFEDCBA|        KJIHGFEDCBA|
>        |         |         |                   |
>        +---------+---------+-------------------+
> 
> or something else?

I think the former of these two is TRT.

The reason is simple, although it took me some time to arrive at it:
column numbering.  In a R2L line, columns should be numbered from the
right margin of the window.  Given the fact that a window's dimensions
can be changed at will, and thus the horizontal position of characters
in R2L lines relative to L2R lines can change, there simply isn't any
other reasonable way of numbering columns of a R2L line.  We will need
that anyway for supporting features like indentation and tab stops in
R2L paragraphs.

And since the columns are numbered from the right margin, hscroll,
too, should scroll R2L lines to the right when L2R lines are hscrolled
to the left.  That's the only reasonable way of generalizing the
current unidirectional hscroll, IMO.  It also preserves the semantics
of window-hscroll (modulo the fact that R2L lines are scrolled from
the right margin, not the left one).

Makes sense?

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[martin rudalics]

 > In a R2L line, columns should be numbered from the
 > right margin of the window.

Technically columns are numbered by counting characters following the
last newline in the buffer text.  Just that for R2L text you have to
jump to the end of the text first and then go back in the text.  Anyway,
column numbers are in the buffer text, unrelated to display or windows.
So I suppose you're using the term window margin as metaphor here.

 > Given the fact that a window's dimensions
 > can be changed at will, and thus the horizontal position of characters
 > in R2L lines relative to L2R lines can change, there simply isn't any
 > other reasonable way of numbering columns of a R2L line.  We will need
 > that anyway for supporting features like indentation and tab stops in
 > R2L paragraphs.

There is one thing I conceptually don't understand yet: When I currently
shrink the width of a window Emacs doesn't hide text at the left side of
the window but text at the right side may get truncated when the window
becomes too small.  Will this behavior be reverted for R2L text in the
sense that text at the left side is truncated?

 > And since the columns are numbered from the right margin, hscroll,
 > too, should scroll R2L lines to the right when L2R lines are hscrolled
 > to the left.  That's the only reasonable way of generalizing the
 > current unidirectional hscroll, IMO.  It also preserves the semantics
 > of window-hscroll (modulo the fact that R2L lines are scrolled from
 > the right margin, not the left one).
 >
 > Makes sense?

Sounds simple and robust.

martin

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[Miles Bader]

martin rudalics <rudalics@gmx.at> writes:
> Technically columns are numbered by counting characters following the
> last newline in the buffer text.

Eh?  Clearly they're not, because things like tabs expand into multiple
columns...  Columns are a display concept, not a buffer concept.

-Miles

-- 
Electricity, n. The cause of all natural phenomena not known to be caused by
something else.

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[Eli Zaretskii]

> From: Miles Bader <miles@gnu.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-bidi@gnu.org,  emacs-devel@gnu.org
> Date: Mon, 01 Feb 2010 06:53:21 +0900
> 
> martin rudalics <rudalics@gmx.at> writes:
> > Technically columns are numbered by counting characters following the
> > last newline in the buffer text.
> 
> Eh?  Clearly they're not, because things like tabs expand into multiple
> columns...  Columns are a display concept, not a buffer concept.

I guess Martin meant to say that column numbering is defined even for
parts of a buffer that are not displayed, or even for buffers that are
not displayed in any window.

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[martin rudalics]

 >> Technically columns are numbered by counting characters following the
 >> last newline in the buffer text.
 >
 > Eh?  Clearly they're not, because things like tabs expand into multiple
 > columns...  Columns are a display concept, not a buffer concept.

Expanding tabs does not depend on displaying the surrounding text but
looks into the variable `tab-width' which is buffer-local and not
related in any sense to display.  Also, you can call a function like
`move-to-column' without ever displaying the associated buffer.

Or, for example, put an overlay with some string as a before-string
property into a buffer and display that buffer so the overlay is shown.
The characters of the overlay are displayed but the columns they occupy
on screen are not counted.

But you're right in that for non-displayable characters or characters
displayed as octals, columns do not count from the last newline either.
And the column numbers supplied by some compilers together with warnings
or error messages are mostly pretty useless for finding the corresponding
position in the source.

So I think we should be able to meet somewhere in the middle ;-)

Anyway - the point I wanted to make was that the number of a column does
not depend on the size of the window displaying the character at that
column or whether text in that window has been scrolled horizontally.

martin

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[Eli Zaretskii]

> Date: Sun, 31 Jan 2010 21:01:29 +0100
> From: martin rudalics <rudalics@gmx.at>
> CC: emacs-bidi@gnu.org, emacs-devel@gnu.org
> 
>  > In a R2L line, columns should be numbered from the
>  > right margin of the window.
> 
> Technically columns are numbered by counting characters following the
> last newline in the buffer text.

As Miles points out, it's not a simple character counting, but I see
what you mean.

> Just that for R2L text you have to jump to the end of the text first
> and then go back in the text.

Actually, if a R2L line displays text made mostly of strong R
characters (which is normally the case), the first character at the
right margin of the window is also the first character following the
newline in the buffer's logical order.


> So I suppose you're using the term window margin as metaphor here.

Yes, in a sense.  In the displayed portion of the buffer, that is
really the window margin, though.

> There is one thing I conceptually don't understand yet: When I currently
> shrink the width of a window Emacs doesn't hide text at the left side of
> the window but text at the right side may get truncated when the window
> becomes too small.  Will this behavior be reverted for R2L text in the
> sense that text at the left side is truncated?

Yes, the truncation for R2L lines will happen at the left, and the
truncation glyph will be displayed there as well.  (It actually
already works like that, at least on a tty.)

>  > And since the columns are numbered from the right margin, hscroll,
>  > too, should scroll R2L lines to the right when L2R lines are hscrolled
>  > to the left.  That's the only reasonable way of generalizing the
>  > current unidirectional hscroll, IMO.  It also preserves the semantics
>  > of window-hscroll (modulo the fact that R2L lines are scrolled from
>  > the right margin, not the left one).
>  >
>  > Makes sense?
> 
> Sounds simple and robust.

Thanks for the feedback.

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[martin rudalics]

 > Actually, if a R2L line displays text made mostly of strong R
 > characters (which is normally the case), the first character at the
 > right margin of the window is also the first character following the
 > newline in the buffer's logical order.

Unless it's horizontally scrolled.

 >> There is one thing I conceptually don't understand yet: When I currently
 >> shrink the width of a window Emacs doesn't hide text at the left side of
 >> the window but text at the right side may get truncated when the window
 >> becomes too small.  Will this behavior be reverted for R2L text in the
 >> sense that text at the left side is truncated?
 >
 > Yes, the truncation for R2L lines will happen at the left, and the
 > truncation glyph will be displayed there as well.  (It actually
 > already works like that, at least on a tty.)

It might be probably interesting for a user to dynamically switch during
typing whether a buffer should be dominantly displayed as R2L or L2R.

martin

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[Eli Zaretskii]

> Date: Mon, 01 Feb 2010 09:34:53 +0100
> From: martin rudalics <rudalics@gmx.at>
> CC: emacs-bidi@gnu.org, emacs-devel@gnu.org
> 
> It might be probably interesting for a user to dynamically switch during
> typing whether a buffer should be dominantly displayed as R2L or L2R.

I already implemented such a feature: a per-buffer variable that
forces all paragraphs to be either L2R or R2L.  A value of `nil' means
the direction of each paragraph is dynamically determined by applying
the rules described in the Unicode Standard Annex 9 (UAX#9).

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[martin rudalics]

 > I already implemented such a feature: a per-buffer variable that
 > forces all paragraphs to be either L2R or R2L.  A value of `nil' means
 > the direction of each paragraph is dynamically determined by applying
 > the rules described in the Unicode Standard Annex 9 (UAX#9).

I meant a function which does (1) set such a variable and (2) apply it
to one or all windows showing a buffer.  Calling this function would
temporarily override any L2R/R2L specifications specified for a file,
buffer, or paragraph.

BTW, do UAX#9 paragraphs require new definitions for `paragraph-start'
or `paragraph-separate'?

martin

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[Eli Zaretskii]

> Date: Tue, 02 Feb 2010 09:08:40 +0100
> From: martin rudalics <rudalics@gmx.at>
> CC: emacs-bidi@gnu.org, emacs-devel@gnu.org
> 
>  > I already implemented such a feature: a per-buffer variable that
>  > forces all paragraphs to be either L2R or R2L.  A value of `nil' means
>  > the direction of each paragraph is dynamically determined by applying
>  > the rules described in the Unicode Standard Annex 9 (UAX#9).
> 
> I meant a function which does (1) set such a variable

You mean, besides "M-x set-variable RET"?

> and (2) apply it to one or all windows showing a buffer.

Currently, the variable is per-buffer, so it affects all the windows
showing that buffer.  Why would one need to do that only in some
windows showing a buffer?

> Calling this function would temporarily override any L2R/R2L
> specifications specified for a file, buffer, or paragraph.

There are no specifications for a file (unless you set the variable
I'm talking about in file's local variables section).  As for
individual paragraphs, control of their base direction is not by some
Emacs setting, but by inserting special formatting characters at the
beginning of each paragraph.  These characters (LRM and RLM) are
supposed to be invisible by default, i.e. displayed as zero-width
space, but they have strong directionality, L for LRM and R for RLM.
Since UAX#9 says that a paragraph's base direction is determined by
its first strong directional character, each one of these two
characters sets the paragraph direction according to directionality of
the character.

It would be easy enough to write a command that inserts LRM or RLM at
the beginning of each paragraph in a buffer or region.  But that's
application level, and I still have a lot of turf to cover before I
get to that.

> BTW, do UAX#9 paragraphs require new definitions for `paragraph-start'
> or `paragraph-separate'?

It does:

   Paragraphs are divided by the Paragraph Separator or appropriate
   Newline Function [...].  Paragraphs may also be determined by
   higher-level protocols: for example, the text in two different
   cells of a table will be in different paragraphs.

and the table of Bidirectional Character Types says that a Paragaraph
Separator type is assigned to the following characters:

   Paragraph separator, appropriate Newline Functions, higher-level
   protocol paragraph determination

Accordingly, in the Unicode Database, the characters CR and LF
(a.k.a. NL) that normally separate lines have the Paragraph Separator
(B) type.

This could sound like a disaster (each line being a separate
paragraph), since Emacs uses hard newlines to fill paragraphs.
Fortunately, UAX#9 leaves a fire escape: it says (see above) that
paragraphs can also be determined by ``higher-level protocols''.  I
used this fire escape to preserve the normal Emacs notion of a
paragraph, including the usual sense of `paragraph-start' and
`paragraph-separate'.  For instance the code that determines the base
direction of each paragraph looks back for a position that matches
`paragraph-start', and then finds the first strong directional
character after that.

So UAX#9 does define a default for paragraph start that is different
from Emacs, but gives us a way to preserve ours.  Which we did.

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[martin rudalics]

 >> I meant a function which does (1) set such a variable
 >
 > You mean, besides "M-x set-variable RET"?

Yes, because it would have affected the appearance of text in one window
only.

 >> and (2) apply it to one or all windows showing a buffer.
 >
 > Currently, the variable is per-buffer, so it affects all the windows
 > showing that buffer.  Why would one need to do that only in some
 > windows showing a buffer?

I thought about giving users the possibility to work with a L2R
dominated view in one window and a R2L dominated one in another.  But if
I now understand your lock-step proposal correctly, these windows would
show the text in a similar fashion anyway.  So there seems no need to do
what I said.

martin

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[Richard Stallman]

    There is one thing I conceptually don't understand yet: When I currently
    shrink the width of a window Emacs doesn't hide text at the left side of
    the window but text at the right side may get truncated when the window
    becomes too small.  Will this behavior be reverted for R2L text in the
    sense that text at the left side is truncated?

Since R2L text lines start from the right margin, that will remain
true after changes in window width.  This is true independent of how
hscroll works.

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[martin rudalics]

 >     There is one thing I conceptually don't understand yet: When I currently
 >     shrink the width of a window Emacs doesn't hide text at the left side of
 >     the window but text at the right side may get truncated when the window
 >     becomes too small.  Will this behavior be reverted for R2L text in the
 >     sense that text at the left side is truncated?
 >
 > Since R2L text lines start from the right margin, that will remain
 > true after changes in window width.  This is true independent of how
 > hscroll works.

With L2R and R2L lines in one and the same buffer we have to decide
which line type is given preference whenever shrinking a window
truncates at least one L2R and at least one R2L line.  "Giving
preference" here means which line type is allowed to place its first
column at the respective window margin.  Usually, the preferred line
type can be derived from the position of `point'.  For example, when
`point' is at the beginning of a R2L line it's appropriate to show the
right margin when shrinking the associated window.

martin

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[tomas]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Tue, Feb 02, 2010 at 09:08:16AM +0100, martin rudalics wrote:
> >     There is one thing I conceptually don't understand yet: When I 
> currently
> >     shrink the width of a window Emacs doesn't hide text at the left side 
> of
> >     the window but text at the right side may get truncated when the 
> window
> >     becomes too small.  Will this behavior be reverted for R2L text in 
> the
> >     sense that text at the left side is truncated?
> >
> > Since R2L text lines start from the right margin, that will remain
> > true after changes in window width.  This is true independent of how
> > hscroll works.
>
> With L2R and R2L lines in one and the same buffer we have to decide
> which line type is given preference whenever shrinking a window
> truncates at least one L2R and at least one R2L line.

(AFAIU) not if you follow Eli's approach of "moving from center" (i.e.
anchor R2L lines right and L2R lines left (which looks as the most
compelling approach to me [1]). That would be "inverted scrolling" in
Stefan's classification in this thread.

There seem to be some limitations to this model, like what to do when
L2R and R2L mix in a line (especially in structured lines, like table
rows, cf. Ehud's post in this thread).

> preference" here means which line type is allowed to place its first
> column at the respective window margin.  Usually, the preferred line
> type can be derived from the position of `point'.  For example, when
> `point' is at the beginning of a R2L line it's appropriate to show the
> right margin when shrinking the associated window.

It seems to me that you'd prefer to keep R2L and L2R texts rigidly tied
(column-wise), if i understand you correctly (i.e. Stefan's "rigid
scrolling"). But I don't know whether this is well-defined without
pre-supposing some arbitrary text with to begin with (and what do you do
with proportional fonts? Ligatures?).

Regards

[1] Disclaimer: I do follow those bidi discussions, but my headd spins
so fast that I don't think my judgements are very valid at the moment.

- -- tomás
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFLaCdFBcgs9XrR2kYRAiTAAJwNElz0qvqa5hcDugsFJ/OIwEtVBgCcCfxy
Q/RkeN+M0uRji5SH0VUYy2U=
=DnPm
-----END PGP SIGNATURE-----

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[martin rudalics]

 >> With L2R and R2L lines in one and the same buffer we have to decide
 >> which line type is given preference whenever shrinking a window
 >> truncates at least one L2R and at least one R2L line.
 >
 > (AFAIU) not if you follow Eli's approach of "moving from center" (i.e.
 > anchor R2L lines right and L2R lines left (which looks as the most
 > compelling approach to me [1]). That would be "inverted scrolling" in
 > Stefan's classification in this thread.

Indeed.  Eli's approach handles this.

 > It seems to me that you'd prefer to keep R2L and L2R texts rigidly tied
 > (column-wise), if i understand you correctly (i.e. Stefan's "rigid
 > scrolling").

I don't prefer that.  But what you say here is the correct conclusion
from what I wrote.

 > [1] Disclaimer: I do follow those bidi discussions, but my headd spins
 > so fast that I don't think my judgements are very valid at the moment.

They are valid.  My head spins faster than yours.

martin

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[Eli Zaretskii]

> Date: Tue, 2 Feb 2010 14:23:17 +0100
> From: tomas@tuxteam.de
> 
> > With L2R and R2L lines in one and the same buffer we have to decide
> > which line type is given preference whenever shrinking a window
> > truncates at least one L2R and at least one R2L line.
> 
> (AFAIU) not if you follow Eli's approach of "moving from center" (i.e.
> anchor R2L lines right and L2R lines left (which looks as the most
> compelling approach to me [1]). That would be "inverted scrolling" in
> Stefan's classification in this thread.

Yes, that's right.

> There seem to be some limitations to this model, like what to do when
> L2R and R2L mix in a line (especially in structured lines, like table
> rows, cf. Ehud's post in this thread).

I see no limitations in that case, and I will explain that in a
moment.

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[tomas]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Tue, Feb 02, 2010 at 09:32:22PM +0200, Eli Zaretskii wrote:
[...]
> > There seem to be some limitations to this model [inverted scrolling]

> I see no limitations in that case, and I will explain that in a
> moment.

Thanks, Eli. I think I see now.

Regards
- -- tomás
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFLbQxdBcgs9XrR2kYRAr9DAJ9qT/BOrMxhOzJD09XRse3axCpBiwCfUoUp
BOw9vZdHjOQ8g1CzeiLGQOM=
=hPq0
-----END PGP SIGNATURE-----

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[Richard Stallman]

    It seems to me that you'd prefer to keep R2L and L2R texts rigidly tied
    (column-wise), if i understand you correctly (i.e. Stefan's "rigid
    scrolling"). But I don't know whether this is well-defined without
    pre-supposing some arbitrary text with to begin with (and what do you do
    with proportional fonts? Ligatures?).

Rigid hscrolling is well defined with proportional fonts.

I don't think Emacs tries to do hscroll properly with proportional
fonts even now with all text L2R.  It would be good to implement this.
Handling R2L text too in a rigid way would not be much harder.

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[tomas]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Tue, Feb 02, 2010 at 04:21:02PM -0500, Richard Stallman wrote:

[...]

> Rigid hscrolling is well defined with proportional fonts.

but it seems to me it would interact badly with window width changes?

(note that my former disclaimer in the mail to Eli still applies. I'm
far from expert in that matter, and whereas I once learnt an R2L
language, even that is very rusty).

> I don't think Emacs tries to do hscroll properly with proportional
> fonts even now with all text L2R.  It would be good to implement this.

That would be nice, yes. AFAIR there was an attempt to use floats as
window positions?

> Handling R2L text too in a rigid way would not be much harder.

See above. As surprising as Eli's approach seems at first, it looks more
convincing to me each time I think about it.

Regards
- -- tomás
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFLbQ2xBcgs9XrR2kYRArYaAJwIBwylgY107ZQ6c7evJuNHAuH+YACfclDh
61KfyUEsPSXjY8eK/L4zbQc=
=biC+
-----END PGP SIGNATURE-----

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[David De La Harpe Golden]

tomas@tuxteam.de wrote:

> 
>> Rigid hscrolling is well defined with proportional fonts.
> 
> but it seems to me it would interact badly with window width changes?
> 

I did outline something that should (I think) deal with that aspect in 
particular in a post last month, but Eli did point out it would still 
necessitate further nontrivial changes to display code - he's got a lot 
on his plate already (understatement) -see:
Message-ID: <4B659BA3.9050406@harpegolden.net>
http://lists.gnu.org/archive/html/emacs-devel/2010-01/msg01558.html

My minor concern is that if the inverted hscrolling method is 
implemented and an emacs is formally released with it, then a subsequent 
implementation of rigid (including window-width-aware rigid) hscrolling 
might also have to deal with offering the inverted hscrolling behaviour 
some users have by then become accustomed to as an option.  That would 
likely be more horribly complicated than only supporting the one type of 
hscrolling.  It's premature to worry about, though, there'll presumably 
be a period with bidi code public in branch and then trunk, but not an 
emacs release.

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[Richard Stallman]

    > Rigid hscrolling is well defined with proportional fonts.

    but it seems to me it would interact badly with window width changes?

Could you be more specific?  I do not see why there would be any
difficulty in this interaction.

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[Richard Stallman]

     > Since R2L text lines start from the right margin, that will remain
     > true after changes in window width.  This is true independent of how
     > hscroll works.

    With L2R and R2L lines in one and the same buffer we have to decide
    which line type is given preference whenever shrinking a window
    truncates at least one L2R and at least one R2L line.  "Giving
    preference" here means which line type is allowed to place its first
    column at the respective window margin.

I think you are right, when the window shrinks that much.
However, if it shrinks but not enough to make both types truncate,
then there is no need to make a choice.

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[David De La Harpe Golden]

[-- Attachment #1: Type: text/plain, Size: 459 bytes --]

Eli Zaretskii wrote:


>  But Emacs does not
> have a rigid line length, so flushing the R2L lines against the right
> margin of the window makes their position depend on the window width.
> Is that okay?  (I think it is.)


FWIW: Yudit (the bidi text editor) uses right edge of window for R2L as 
you suggest (and left edge for L2R), reflowing as you resize the window. 
It apparently only line wraps though, no truncation, so no horizontal 
scrolling.






[-- Attachment #2: yudit_bidi.png --]
[-- Type: image/png, Size: 10332 bytes --]

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[Richard Stallman]

    Let me start with an example of 2 single-line paragraphs (the frame
    around the text shows the window borders):

	  +---------------------------------------+
	  |abcdefghijk                            |
	  |                            KJIHGFEDCBA|
	  |                                       |
	  +---------------------------------------+

Why are there 28 spaces; what makes KJIHGFEDCBA end at column 39?
Is that determined by the width of the window, or by some separate
setting analogous to fill-column?

    Looks okay, right?  But what if I split the window horizontally (with
    "C-x 3"):

	  +-------------------+-------------------+
	  |abcdefghijk        |abcdefghijk        |
	  |        KJIHGFEDCBA|        KJIHGFEDCBA|
	  |                   |                   |
	  +-------------------+-------------------+

    Does this look correct?  Is anyone bothered by the fact that the
    relative horizontal position of the two lines has changed, and yet
    there are no line truncation glyphs anywhere in sight?

I think users will be happy with it.  If the column where KJIHGFEDCBA
ends is determined by the width, then this is the behavior that follows
the rule.  Anything else would be inconsistent.

      But Emacs does not
    have a rigid line length, so flushing the R2L lines against the right
    margin of the window makes their position depend on the window width.
    Is that okay?  (I think it is.)

Maybe there SHOULD be a specified line length for this purpose.
(Perhaps fill-column?  Or "use fill-column" could be one of the
settings.)  nil could mean "use the window width".

When the width is specified, not nil, then the right behavior
is to use that width even if it causes truncation.

    Now the crucial question: what do you think should happen if I scroll
    the leftmost window horizontally so that the end of the first line
    (the characters "ijk") become visible?  I think it should be this:

	  +---------+---------+-------------------+
	  |$ijk     |abcdefgh$|abcdefghijk        |
	  |$HGFEDCBA|$HGFEDCBA|        KJIHGFEDCBA|
	  |         |         |                   |
	  +---------+---------+-------------------+

Horizontal scrolling should apply to the whole window contents
in a rigid fashion.

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[David De La Harpe Golden]

Richard Stallman wrote:


> Maybe there SHOULD be a specified line length for this purpose.
> (Perhaps fill-column?  Or "use fill-column" could be one of the
> settings.)  nil could mean "use the window width".
> 
>     Now the crucial question: what do you think should happen if I scroll
>     the leftmost window horizontally so that the end of the first line
>     (the characters "ijk") become visible?  I think it should be this:
> 
> 	  +---------+---------+-------------------+
> 	  |$ijk     |abcdefgh$|abcdefghijk        |
> 	  |$HGFEDCBA|$HGFEDCBA|        KJIHGFEDCBA|
> 	  |         |         |                   |
> 	  +---------+---------+-------------------+
> 
> Horizontal scrolling should apply to the whole window contents
> in a rigid fashion.
> 
> 

Rigid-  You mean, when l2r lines scroll right, r2l lines scroll right
- not mirrored and not not-scrolled?  That's not a problem with
a fixed column, but how to reconcile that with using the flexible window 
right margin as the r2l flush (which, having tried yudit, is more
pleasant IMO)?

Can hscroll amount ever be less than window-width increments (I seldom 
use it, preferring wrapping)? Yes, yes it can, just depends on 
hscroll-step settings.

So maybe the following would work?

I think it might produce unsurprising  "rigid-body-motion" type 
scrolling results, yet still allow use of the window right margin:

When you have scrolled horizontally, (window-hscroll) positive, leave a 
gap from the right window margin for r2l about equal to the amount l2r 
lines have hscrolled.   (And do the same for l2r lines when long r2l 
lines are  scrolled horizontally, using the suggestion of negative 
values of (window-hscroll) for r2l leftward scrolling...)

That may sound odd, put that way, so let's illustrate. I'm going
to switch to fringe truncation indicators to simplify things, otherwise
my tiny brain gets confused:

  	  +----------+--------- +--------------------+
  	  |:abcdefgh$|:abcdefgh$|:abcdefghijk       :|
  	  |$HGFEDCBA:|$HGFEDCBA:|:       KJIHGFEDCBA:|
  	  |:        :|:        :|:                  :|
  	  +----------+----------+--------------------+

  	  +----------+--------- +--------------------+
  	  |$ijk     :|:abcdefgh$|:abcdefghijk       :|
  	  |$........:|$HGFEDCBA:|:       KJIHGFEDCBA:|
  	  |:        :|:        :|:                  :|
  	  +----------+----------+--------------------+


.... is an amount equal to the horizontal scroll.

Okay, that was a whole window width scroll, not very clear, since that 
would mean the r2l bit just disappears. Let's try again with a smaller 
window width hscroll.  Note how r2l letters that were under particular 
l2r letters still line up for a given window width. That could be 
desirable, especially for visual line movement.


  	  +----------+--------- +--------------------+
  	  |:abcdefgh$|:abcdefgh$|:abcdefghijk       :|
  	  |$HGFEDCBA:|$HGFEDCBA:|:       KJIHGFEDCBA:|
  	  |:        :|:        :|:                  :|
  	  +----------+----------+--------------------+

  	  +----------+--------- +--------------------+
  	  |$cdefghij$|:abcdefgh$|:abcdefghijk       :|
  	  |$FEDCBA..:|$HGFEDCBA:|:       KJIHGFEDCBA:|
  	  |:        :|:        :|:                  :|
  	  +----------+----------+--------------------+

  	  +----------+--------- +--------------------+
  	  |$efghijk :|:abcdefgh$|:abcdefghijk       :|
  	  |$DCBA....:|$HGFEDCBA:|:       KJIHGFEDCBA:|
  	  |:        :|:        :|:                  :|
  	  +----------+----------+--------------------+

  	  +----------+--------- +--------------------+
  	  |$ghijk   :|:abcdefgh$|:abcdefghijk       :|
  	  |$BA......:|$HGFEDCBA:|:       KJIHGFEDCBA:|
  	  |:        :|:        :|:                  :|
  	  +----------+----------+--------------------+

  	  +----------+--------- +--------------------+
  	  |$ijk     :|:abcdefgh$|:abcdefghijk       :|
  	  |$........:|$HGFEDCBA:|:       KJIHGFEDCBA:|
  	  |:        :|:        :|:                  :|
  	  +----------+----------+--------------------+


Let's try the other direction for a truncated r2l line in the 2nd 
window, presumably a negative (window-hscroll) amount:

  	  +----------+--------- +--------------------+
  	  |:abcdefgh$|:abcdefgh$|:abcdefghijk       :|
  	  |$HGFEDCBA:|$HGFEDCBA:|:       KJIHGFEDCBA:|
  	  |:        :|:        :|:                  :|
  	  +----------+----------+--------------------+

  	  +----------+--------- +--------------------+
  	  |$cdefghij$|:,,abcdef$|:abcdefghijk       :|
  	  |$FEDCBA..:|$JIHGFEDC$|:       KJIHGFEDCBA:|
  	  |:        :|:        :|:                  :|
  	  +----------+----------+--------------------+

  	  +----------+--------- +--------------------+
  	  |$efghijk :|:,,,,abcd$|:abcdefghijk       :|
  	  |$DCBA....:|: KJIHGFE$|:       KJIHGFEDCBA:|
  	  |:        :|:        :|:                  :|
  	  +----------+----------+--------------------+

  	  +----------+--------- +--------------------+
  	  |$ghijk   :|:,,,,,,ab$|:abcdefghijk       :|
  	  |$BA......:|:   KJIHG$|:       KJIHGFEDCBA:|
  	  |:        :|:        :|:                  :|
  	  +----------+----------+--------------------+

  	  +----------+--------- +--------------------+
  	  |$ijk     :|:,,,,,,,,$|:abcdefghijk       :|
  	  |$........:|:     KJI$|:       KJIHGFEDCBA:|
  	  |:        :|:        :|:                  :|
  	  +----------+----------+--------------------+

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[Eli Zaretskii]

> Date: Sun, 31 Jan 2010 15:02:59 +0000
> From: David De La Harpe Golden <david@harpegolden.net>
> CC: Eli Zaretskii <eliz@gnu.org>, emacs-bidi@gnu.org, 
>  emacs-devel@gnu.org
> 
>   	  +----------+--------- +--------------------+
>   	  |:abcdefgh$|:abcdefgh$|:abcdefghijk       :|
>   	  |$HGFEDCBA:|$HGFEDCBA:|:       KJIHGFEDCBA:|
>   	  |:        :|:        :|:                  :|
>   	  +----------+----------+--------------------+
> 
>   	  +----------+--------- +--------------------+
>   	  |$cdefghij$|:abcdefgh$|:abcdefghijk       :|
>   	  |$FEDCBA..:|$HGFEDCBA:|:       KJIHGFEDCBA:|
>   	  |:        :|:        :|:                  :|
>   	  +----------+----------+--------------------+
> 
>   	  +----------+--------- +--------------------+
>   	  |$efghijk :|:abcdefgh$|:abcdefghijk       :|
>   	  |$DCBA....:|$HGFEDCBA:|:       KJIHGFEDCBA:|
>   	  |:        :|:        :|:                  :|
>   	  +----------+----------+--------------------+
> 
>   	  +----------+--------- +--------------------+
>   	  |$ghijk   :|:abcdefgh$|:abcdefghijk       :|
>   	  |$BA......:|$HGFEDCBA:|:       KJIHGFEDCBA:|
>   	  |:        :|:        :|:                  :|
>   	  +----------+----------+--------------------+
> 
>   	  +----------+--------- +--------------------+
>   	  |$ijk     :|:abcdefgh$|:abcdefghijk       :|
>   	  |$........:|$HGFEDCBA:|:       KJIHGFEDCBA:|
>   	  |:        :|:        :|:                  :|
>   	  +----------+----------+--------------------+

This may be a good feature, but it's a new feature, and would require
non-trivial changes in how the display engine lays out glyphs in the
so-called ``glyph row'' (which is a generalization of a line).  I
think I want to avoid that for now, as my job is huge even without
that.

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> CC: emacs-devel@gnu.org, emacs-bidi@gnu.org
> Date: Sun, 31 Jan 2010 07:42:09 -0500
> 
> 	  +---------------------------------------+
> 	  |abcdefghijk                            |
> 	  |                            KJIHGFEDCBA|
> 	  |                                       |
> 	  +---------------------------------------+
> 
> Why are there 28 spaces; what makes KJIHGFEDCBA end at column 39?

The fact that there are only 11 characters in that line, and the fact
that the window I drew happens to be 39 characters wide.  An R2L line
is drawn with its first character (A) flushed to the right margin of
the window, the second character (B) one column to the left of it,
etc.

> Is that determined by the width of the window, or by some separate
> setting analogous to fill-column?

Only the width of the window.

>       But Emacs does not
>     have a rigid line length, so flushing the R2L lines against the right
>     margin of the window makes their position depend on the window width.
>     Is that okay?  (I think it is.)
> 
> Maybe there SHOULD be a specified line length for this purpose.
> (Perhaps fill-column?  Or "use fill-column" could be one of the
> settings.)  nil could mean "use the window width".
> 
> When the width is specified, not nil, then the right behavior
> is to use that width even if it causes truncation.

You are talking about introducing a new feature.  That could be good,
but for now I'd like to have a bidi display without new features.  We
will need that anyway, even under your suggestion, for supporting the
nil value.

>     Now the crucial question: what do you think should happen if I scroll
>     the leftmost window horizontally so that the end of the first line
>     (the characters "ijk") become visible?  I think it should be this:
> 
> 	  +---------+---------+-------------------+
> 	  |$ijk     |abcdefgh$|abcdefghijk        |
> 	  |$HGFEDCBA|$HGFEDCBA|        KJIHGFEDCBA|
> 	  |         |         |                   |
> 	  +---------+---------+-------------------+
> 
> Horizontal scrolling should apply to the whole window contents
> in a rigid fashion.

But that cannot be done in this case, because the second line cannot
be scrolled to the left without moving its first character A from the
right margin.

I now think that the following display is the right one (see my other
message for the reasons):

          +---------+---------+-------------------+
          |$ijk     |abcdefgh$|abcdefghijk        |
          |     KJI$|$HGFEDCBA|        KJIHGFEDCBA|
          |         |         |                   |
          +---------+---------+-------------------+

and if the window is scrolled by 2 characters less, then it should
look like this:

          +---------+---------+-------------------+
          |$ghijk   |abcdefgh$|abcdefghijk        |
          |   KJIHG$|$HGFEDCBA|        KJIHGFEDCBA|
          |         |         |                   |
          +---------+---------+-------------------+

etc.

The principle is: the only point of reference is the margin, the left
one for L2R lines, and the right one for R2L.

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[Richard Stallman]

 
    > Maybe there SHOULD be a specified line length for this purpose.
    > (Perhaps fill-column?  Or "use fill-column" could be one of the
    > settings.)  nil could mean "use the window width".
    > 
    > When the width is specified, not nil, then the right behavior
    > is to use that width even if it causes truncation.

    You are talking about introducing a new feature.  That could be good,
    but for now I'd like to have a bidi display without new features.

Ok, let's stick to the case where the window width controls it.

    > Horizontal scrolling should apply to the whole window contents
    > in a rigid fashion.

    But that cannot be done in this case, because the second line cannot
    be scrolled to the left without moving its first character A from the
    right margin.

Horizontal scrolling does whatever we make it do.  The question
is what behavior is best.

I'm saying that horizontal scrolling should move the window contents
rigidly left or right, just as vertical scrolling moves the contents
rigidly up or down.

Perhaps with bidi text we should allow negative hscroll, in other
words, scrolling the whole contents to the left of the usual position.

The definition of hscroll is not the same as how we number columns for
indentation purposes.  Indentation is defined in terms of the order of
text in the buffer.  In a line that is entirely RTL, indentation needs
to start from the right margin and work leftward.  By contrast,
horizontal scrolling is defined in terms of left and right, not
forward and back.

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Date: Mon, 01 Feb 2010 16:05:19 -0500
> Cc: emacs-bidi@gnu.org, emacs-devel@gnu.org
> 
>  
>     > Horizontal scrolling should apply to the whole window contents
>     > in a rigid fashion.
> 
>     But that cannot be done in this case, because the second line cannot
>     be scrolled to the left without moving its first character A from the
>     right margin.
> 
> Horizontal scrolling does whatever we make it do.  The question
> is what behavior is best.
> 
> I'm saying that horizontal scrolling should move the window contents
> rigidly left or right, just as vertical scrolling moves the contents
> rigidly up or down.

That can be done, but then we will probably need to introduce some new
display features to indicate that the first character is scrolled off
the margin.  Otherwise, the display will look confusing wrt where each
line begins.  For example, we could draw special glyphs at the left
margin in L2R lines and on the right margin in R2L lines, or we might
use something similar to trailing-whitespace display.

> The definition of hscroll is not the same as how we number columns for
> indentation purposes.

That's true, but the effect of hscroll is defined in terms of columns,
so I think it's best not to break the relation between column
numbering and hscroll.

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[Richard Stallman]

    That can be done, but then we will probably need to introduce some new
    display features to indicate that the first character is scrolled off
    the margin.

The margin glyphs indicate that; I don't we need to change anything there.

    > The definition of hscroll is not the same as how we number columns for
    > indentation purposes.

    That's true, but the effect of hscroll is defined in terms of columns,
    so I think it's best not to break the relation between column
    numbering and hscroll.

The relation is that they count in the same units.  R2L text will not
alter that relationship in any case; one column as regards scrolling
will still be one column as regards indentation.

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[Ehud Karni]

On Sat, 30 Jan 2010 15:44:28 Eli Zaretskii wrote:
>
> Here's a peculiar design decision that needs to be made for the bidi
> display: how to display horizontally scrolled lines in a buffer with
> mixed L2R and R2L paragraphs.
>
> Let me start with an example of 2 single-line paragraphs (the frame
> around the text shows the window borders):
>
>       +---------------------------------------+
>       |abcdefghijk                            |
>       |                            KJIHGFEDCBA|
>       |                                       |
>       +---------------------------------------+

[sniped]  Eli's final conclusion for scrolled text is:

>       +---------+---------+-------------------+
>       |$ijk     |abcdefgh$|abcdefghijk        |
>       |     KJI$|$HGFEDCBA|        KJIHGFEDCBA|
>       |         |         |                   |
>       +---------+---------+-------------------+

I agree with Eli's decision, but it applies only to non-technical text.

Consider Example1: a list of items, each line has: name, quantity,
catalog-id and description. Most of the names are in Latin but some
are in Hebrew.
In this case there must be an external dictation of paragraph direction.

Example 2: same list as above, all the names are in Latin, the
description is very long, some of the descriptions are in Hebrew.
       +----------------------------------------+
       |name1 1234 catag1 very-long-description |
       |name2 1234 catag2 NOITPIRCSED-GNOL-YREV |
       +----------------------------------------+
and in a smaller window:
       +-------------------------------+
       |name1 1234 catag1 very-long-de$|
       |name2 1234 catag2 ED-GNOL-YREV$|
       +-------------------------------+
scrolled (5) in a smaller window:
       +-------------------------------+
       |$1234 catag1 very-long-descrip$|
       |$1234 catag2 PIRCSED-GNOL-YREV$|
       +--------------------------------
scrolled (23) in a smaller window:
       +-------------------------------+
       |$ong-description               |
       |$NOITPIRCSED-GNO               |
       +--------------------------------

Example 3: text, mostly in Hebrew but some lines have no Hebrew,
like arithmetic formulas (Latin + digits + neutrals only) or
ls output (real recent email I got).

Here we could use either external dictation of RTL direction or
automatic paragraph direction. In RTL direction the formula may be
displayed wrongly (if neutral characters are on the sides). When
automatic paragraph direction is used, short lines may be too far
apart. In this case, something like `right-margin-goal-column' as
suggested by RMS is a good solution.

Ehud.


--
 Ehud Karni           Tel: +972-3-7966-561  /"\
 Mivtach - Simon      Fax: +972-3-7976-561  \ /  ASCII Ribbon Campaign
 Insurance agencies   (USA) voice mail and   X   Against   HTML   Mail
 http://www.mvs.co.il  FAX:  1-815-5509341  / \
 GnuPG: 98EA398D <http://www.keyserver.net/>    Better Safe Than Sorry

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[Eli Zaretskii]

> Date: Mon, 1 Feb 2010 16:00:45 +0200
> From: "Ehud Karni" <ehud@unix.mvs.co.il>
> Cc: emacs-devel@gnu.org, emacs-bidi@gnu.org
> 
> [sniped]  Eli's final conclusion for scrolled text is:
> 
> >       +---------+---------+-------------------+
> >       |$ijk     |abcdefgh$|abcdefghijk        |
> >       |     KJI$|$HGFEDCBA|        KJIHGFEDCBA|
> >       |         |         |                   |
> >       +---------+---------+-------------------+
> 
> I agree with Eli's decision, but it applies only to non-technical text.

What is a ``non-technical text'', and how is it different from the
other kind?  Also, to what kind belong your examples in this message?

> Consider Example1: a list of items, each line has: name, quantity,
> catalog-id and description. Most of the names are in Latin but some
> are in Hebrew.

Where is Example1?  I don't see it in the message you posted?  Did you
talk in general about itemized lists with mixed L2R and R2L text?

> In this case there must be an external dictation of paragraph direction.

There is already a variable to force certain paragraph direction on
all the paragraphs in a buffer.  Finer (per-paragraph) control can be
exercised by starting a paragraph with a suitable mark (RLM or LRM)
character.

But what does this have to do with the issue at hand?

> scrolled (23) in a smaller window:
>        +-------------------------------+
>        |$ong-description               |
>        |$NOITPIRCSED-GNO               |
>        +--------------------------------

Why like above and not like below:

       +-------------------------------+
       |$ong-description               |
       |$RCSED-GNOL-YREV               |
       +--------------------------------

?

> Example 3: text, mostly in Hebrew but some lines have no Hebrew,
> like arithmetic formulas (Latin + digits + neutrals only) or
> ls output (real recent email I got).
> 
> Here we could use either external dictation of RTL direction or
> automatic paragraph direction. In RTL direction the formula may be
> displayed wrongly (if neutral characters are on the sides). When
> automatic paragraph direction is used, short lines may be too far
> apart.

A real example will help here.

> In this case, something like `right-margin-goal-column' as suggested
> by RMS is a good solution.

The problem with such a ``rigid scrolling'' is that it must affect L2R
lines as well, for consistency.  In the current display engine, we
don't allow scrolling to the right (e.g., with C-x >) once column zero
is visible.  With rigid scrolling, we will need to allow it.

Anyway, thanks for the feedback.

Re: [emacs-bidi] Mixed L2R and R2L paragraphs and horizontal scroll

[Ehud Karni]

On Mon, 01 Feb 2010 22:18:31 Eli Zaretskii wrote:
>
> What is a ``non-technical text'', and how is it different from the
> other kind?  Also, to what kind belong your examples in this message?

Non technical text is composed of long sentences of words, mostly in one
language (e.g Man page). Technical text may have less words and more
tokens and non alphabetic symbols (e.g. script, ls output).

> > Consider Example1: a list of items, each line has: name, quantity,
> > catalog-id and description. Most of the names are in Latin but some
> > are in Hebrew.
>
> Where is Example1?  I don't see it in the message you posted?  Did you
> talk in general about itemized lists with mixed L2R and R2L text?

I talked in general. The various lines begin with Hebrew or Latin.

> > In this case there must be an external dictation of paragraph direction.
>
> There is already a variable to force certain paragraph direction on
> all the paragraphs in a buffer.  Finer (per-paragraph) control can be
> exercised by starting a paragraph with a suitable mark (RLM or LRM)
> character.

Yes, I just wanted to emphasize that automatic per-paragraph (default?)
direction may be very wrong some times.

> But what does this have to do with the issue at hand?

Nothing to do with scrolling.

> > scrolled (23) in a smaller window:
> >        +-------------------------------+
> >        |$ong-description               |
> >        |$NOITPIRCSED-GNO               |
> >        +--------------------------------
>
> Why like above and not like below:
>
>        +-------------------------------+
>        |$ong-description               |
>        |$RCSED-GNOL-YREV               |
>        +--------------------------------

Because if you choose your way of scrolling, you always see the same
part (of different length) - the beginning of the Hebrew text.
Suppose the the Hebrew text is very long, twice the window size, then
you'll see only the first width-of-screen Hebrew characters, no matter
how much scrolling you do. I taught that was one of the reasons for
your conclusion of (KJI and not CBA):
ez>     +---------+---------+-------------------+
ez>     |$ijk     |abcdefgh$|abcdefghijk        |
ez>     |     KJI$|$HGFEDCBA|        KJIHGFEDCBA|
ez>     +---------+---------+-------------------+

> > Example 3: text, mostly in Hebrew but some lines have no Hebrew,
> > like arithmetic formulas (Latin + digits + neutrals only) or
> > ls output (real recent email I got).
> >
> > Here we could use either external dictation of RTL direction or
> > automatic paragraph direction. In RTL direction the formula may be
> > displayed wrongly (if neutral characters are on the sides). When
> > automatic paragraph direction is used, short lines may be too far
> > apart.
>
> A real example will help here.

Here is a real example (upper case = Hebrew)
PUT THE FILES IN
/dir1/sdr2/sdr3
AND I WILL TAKE THEM

This will appear as (think of 160 character wide text screen):
                                                   NI SELIF EHT TUP
/dir1/sdr2/sdr3
                                               MEHT EKAT LLIW I DNA


> > In this case, something like `right-margin-goal-column' as suggested
> > by RMS is a good solution.
>
> The problem with such a ``rigid scrolling'' is that it must affect L2R
> lines as well, for consistency.  In the current display engine, we
> don't allow scrolling to the right (e.g., with C-x >) once column zero
> is visible.  With rigid scrolling, we will need to allow it.

I can always split the Emacs window horizontally and change it size
until I have the best fit, It just takes more work.

Ehud.


--
 Ehud Karni           Tel: +972-3-7966-561  /"\
 Mivtach - Simon      Fax: +972-3-7976-561  \ /  ASCII Ribbon Campaign
 Insurance agencies   (USA) voice mail and   X   Against   HTML   Mail
 http://www.mvs.co.il  FAX:  1-815-5509341  / \
 GnuPG: 98EA398D <http://www.keyserver.net/>    Better Safe Than Sorry

Re: [emacs-bidi] Mixed L2R and R2L paragraphs and horizontal scroll

[Eli Zaretskii]

> Date: Tue, 2 Feb 2010 00:05:28 +0200
> From: "Ehud Karni" <ehud@unix.mvs.co.il>
> Cc: emacs-devel@gnu.org, emacs-bidi@gnu.org
> 
> > > scrolled (23) in a smaller window:
> > >        +-------------------------------+
> > >        |$ong-description               |
> > >        |$NOITPIRCSED-GNO               |
> > >        +--------------------------------
> >
> > Why like above and not like below:
> >
> >        +-------------------------------+
> >        |$ong-description               |
> >        |$RCSED-GNOL-YREV               |
> >        +--------------------------------
> 
> Because if you choose your way of scrolling, you always see the same
> part (of different length) - the beginning of the Hebrew text.

Well, actually, no, that's not what I meant.  Although I understand
how I could confuse people (even you ;-) about what I think should
happen.

So let me tell directly and unequivocally what I propose.  There are
two main principles:

  . Line truncation and continuation is done in the visual order

  . Horizontal scrolling is done in the logical order

What this means is that an L2R line that ends with a stretch of R2L
text will be continued as follows:

       +-------------------------------+
       |name2 1234 catag2 NOITPIRCSED-\|
       |GNOL-YREV                      |
       +-------------------------------+

and truncated thusly:

       +-------------------------------+
       |name2 1234 catag2 NOITPIRCSED-$|
       +-------------------------------+

Similarly, an R2L line will be continued like this:

       +-------------------------------+
       |/RCSED-GNOL-YREV-YREV-YREV-YREV|
       |                         NOITPI|
       +-------------------------------+

and truncated like this:

       +-------------------------------+
       |$RCSED-GNOL-YREV-YREV-YREV-YREV|
       +-------------------------------+

Horizontal scrolling will cause these two paragraphs:

       +-------------------------------+
       |name1 1234 catag1 very-long-de$|
       |                               |
       |$RCSED-GNOL-YREV-YREV-YREV-YREV|
       +-------------------------------+

to successively look like below, each time after hscrolling by one or
more columns to the left:

       +-------------------------------+
       |$me1 1234 catag1 very-long-des$|
       |                               |
       |$IRCSED-GNOL-YREV-YREV-YREV-YR$|
       +-------------------------------+

       +-------------------------------+
       |$e1 1234 catag1 very-long-desc$|
       |                               |
       |$PIRCSED-GNOL-YREV-YREV-YREV-Y$|
       +-------------------------------+
...
       +-------------------------------+
       |$ catag1 very-long-description |
       |                               |
       | NOITPIRCSED-GNOL-YREV-YREV-YR$|
       +-------------------------------+
...
       +-------------------------------+
       |$very-long-description         |
       |                               |
       |         NOITPIRCSED-GNOL-YREV$|
       +-------------------------------+

Okay?

At least regarding line truncation, I checked ``prior art''.  The
Wordprocessor-That-Shall-Remain-Unnamed truncates long lines exactly
like I show above: in the visual order.  (Of course, it doesn't have
the truncation glyphs, but instead pops up the horizontal scroll bar.)
My other reason for doing truncation and continuation in visual order
is that it's much easier ;-) The code that produces glyphs and fills
the glyph matrices with them proceeds in the visual order, as it did
in the unidirectional display.  So it is easy enough to stop when we
hit the window margin.  Truncating in logical order, i.e. producing
something like

       +-------------------------------+
       |name2 1234 catag2 ED-GNOL-YREV$|
       +-------------------------------+

requires a much more serious surgery of the redisplay code.

Note that ED-GNOL-YREV in the last example above is the _beginning_ of
the R2L text, as opposed to

       +-------------------------------+
       |name2 1234 catag2 NOITPIRCSED-$|
       +-------------------------------+

in my suggestion, where we display the _end_ of the R2L text.

> Suppose the the Hebrew text is very long, twice the window size, then
> you'll see only the first width-of-screen Hebrew characters, no matter
> how much scrolling you do.

I hope you see now that this is not so.  Scrolling "to the left" will
eventually show every character of the R2L text at the end of the
line.

> > > Example 3: text, mostly in Hebrew but some lines have no Hebrew,
> > > like arithmetic formulas (Latin + digits + neutrals only) or
> > > ls output (real recent email I got).
> > >
> > > Here we could use either external dictation of RTL direction or
> > > automatic paragraph direction. In RTL direction the formula may be
> > > displayed wrongly (if neutral characters are on the sides). When
> > > automatic paragraph direction is used, short lines may be too far
> > > apart.
> >
> > A real example will help here.
> 
> Here is a real example (upper case = Hebrew)
> PUT THE FILES IN
> /dir1/sdr2/sdr3
> AND I WILL TAKE THEM
> 
> This will appear as (think of 160 character wide text screen):
>                                                    NI SELIF EHT TUP
> /dir1/sdr2/sdr3
>                                                MEHT EKAT LLIW I DNA

And why would ``rigid scrolling'' help in this case?

Re: [emacs-bidi] Mixed L2R and R2L paragraphs and horizontal scroll

[Ehud Karni]

On Tue, 02 Feb 2010 22:04:51 Eli Zaretskii wrote:
>
> So let me tell directly and unequivocally what I propose.  There are
> two main principles:
>
>   . Line truncation and continuation is done in the visual order
>
>   . Horizontal scrolling is done in the logical order

What you describe here is the Emacs screen as rectangle frame moving
over the visual ordered text. This is technically sound but very wrong
from a user view point. The reason is that Hebrew reader see the
"continuation" lines in reverse order, and has to read this from the
last (continuation) line (the most down one) upward.

No word processor that uses that approach (check open-office with a
long enough text, or with the other OS word processor).

More reasons to use a different strategy below.

> What this means is that an L2R line that ends with a stretch of R2L
> text will be continued as follows:
>
>        +-------------------------------+
>        |name2 1234 catag2 NOITPIRCSED-\|
>        |GNOL-YREV                      |
>        +-------------------------------+

See the problem - the user must start reading from the second line.

> and truncated thusly:
>
>        +-------------------------------+
>        |name2 1234 catag2 NOITPIRCSED-$|
>        +-------------------------------+

The truncation is technically correct but wrong from user perspective.


> Similarly, an R2L line will be continued like this:
>
>        +-------------------------------+
>        |/RCSED-GNOL-YREV-YREV-YREV-YREV|
>        |                         NOITPI|
>        +-------------------------------+
>
> and truncated like this:
>
>        +-------------------------------+
>        |$RCSED-GNOL-YREV-YREV-YREV-YREV|
>        +-------------------------------+

This is the correct way.

In the your following examples you omitted the long Hebrew text within
a L2R paragraph, which is the problematic case. Let me try to show how
I expect it to work (in continuation/scrolling situation).

The logical text is "some latin text followed by HEBREW VERY1 VERY2
LONG TEXT FOR DEMONSTRATING OF THE DIFFERENT SCROLLING OPTIONS AND
THE UNNATURAL RESULT OF ELIS APPROACH WITH small latin tail"

With your approach it will appear as (first, wider screen):

       +----------------------------------------+
       |some latin text followed by HTIW HCAORP$|
       |PA SILE FO TLUSER LARUTANNU EHT DNA SNO$|
       |ITPO GNILLORCS TNEREFFID EHT FO GNITART$|
       |SNOMED ROF TXET GNOL 2YREV 1YREV WERBEH$|
       | small latin tail                       |
       +----------------------------------------+

The Hebrew starts at 4th visual lines and goes upward.

with smaller screen (assume only 7 lines long):

       +--------------------+
       |some latin text fol$|
       |lowed by HTIW HCAOR$|
       |PPA SILE FO TLUSER $|
       |LARUTANNU EHT DNA S$|
       |NOITPO GNILLORCS TN$|
       |EREFFID EHT FO GNIT$|
       |ARTSNOMED ROF TXET $|
       +--------------------+

Oops, the beginning of the Hebrew text disappeared.

Now if you display the same Hebrew text (without the Latin
prefix) in R2L:

       +----------------------------------------+
       |$SNOMED ROF TXET GNOL 2YREV 1YREV WERBEH|
       |$ITPO GNILLORCS TNEREFFID EHT FO GNITART|
       |$PA SILE FO TLUSER LARUTANNU EHT DNA SNO|
       |            small latin tail HTIW HCAORP|
       +----------------------------------------+

and in smaller screen:

       +--------------------+
       |$ 2YREV 1YREV WERBEH|
       |$NOMED ROF TXET GNOL|
       |$FID EHT FO GNITARTS|
       |$PO GNILLORCS TNEREF|
       |$TANNU EHT DNA SNOIT|
       |$SILE FO TLUSER LARU|
       |$???? HTIW HCAORPPA |
       +--------------------+
The last line ???? are "tail" by Eli's way and "smal" by my way.

See the difference ?  (please do it with real Hebrew and try to
read the Hebrew text according to your approach).

I claim the scrolling should follow the "same" rules.
e.g. 1: wide screen scrolled 20 (half width):
       +----------------------------------------+
       |$owd by OF TXET GNOL 2YREV 1YREV WERBEH$|
       +----------------------------------------+
e.g. 2: wide screen scrolled 60 (1.5 width):
       +----------------------------------------+
       |$LLORCS TNEREFFID EHT FO GNITARTSNOMED $|
       +----------------------------------------+
e.g. 3: wide screen scrolled 150:
       +----------------------------------------+
       |$HTIW H small latin tail                |
       +----------------------------------------+

The word "same" is between quotes because it is the same appearance
even though the rules may be slightly different.

Ehud.


--
 Ehud Karni           Tel: +972-3-7966-561  /"\
 Mivtach - Simon      Fax: +972-3-7976-561  \ /  ASCII Ribbon Campaign
 Insurance agencies   (USA) voice mail and   X   Against   HTML   Mail
 http://www.mvs.co.il  FAX:  1-815-5509341  / \
 GnuPG: 98EA398D <http://www.keyserver.net/>    Better Safe Than Sorry

Re: [emacs-bidi] Mixed L2R and R2L paragraphs and horizontal scroll

[Eli Zaretskii]

> Date: Wed, 3 Feb 2010 15:10:52 +0200
> From: "Ehud Karni" <ehud@unix.mvs.co.il>
> Cc: emacs-bidi@gnu.org, emacs-devel@gnu.org
> 
> On Tue, 02 Feb 2010 22:04:51 Eli Zaretskii wrote:
> >
> > So let me tell directly and unequivocally what I propose.  There are
> > two main principles:
> >
> >   . Line truncation and continuation is done in the visual order
> >
> >   . Horizontal scrolling is done in the logical order
> 
> What you describe here is the Emacs screen as rectangle frame moving
> over the visual ordered text.

For truncation and continuation, yes.  For horizontal scrolling, no:
it works in the logical order.  That is, scroll-left moves text in
each line towards its respective margin: left margin for L2R lines,
right margin for R2L.  More about this near the end of this message.

> No word processor that uses that approach (check open-office with a
> long enough text, or with the other OS word processor).

I didn't try OpenOffice, but that "other OS word processor" does it
exactly like I described, at least with truncation (it does not have
the equivalent of continuation lines, AFAICS).

Maybe you are talking about how lines are re-flowed by these word
processors.  If so, the Emacs equivalent is fill-paragraph, or the
various packages (like longlines) that do it automatically for you.
When a line is filled, a newline is inserted, and then the order of
the text spread between several lines is what you expect:

> > What this means is that an L2R line that ends with a stretch of R2L
> > text will be continued as follows:
> >
> >        +-------------------------------+
> >        |name2 1234 catag2 NOITPIRCSED-\|
> >        |GNOL-YREV                      |
> >        +-------------------------------+
> 
> See the problem - the user must start reading from the second line.

Right.  If you want to read it correctly, use fill-paragraph.  Then
this will display as

        +-------------------------------+
        |name2 1234 catag2 -GNOL-YREV   |
        |NOITPIRCSED                    |
        +-------------------------------+

(That's because bidi reordering stops at the newline and restarts anew
after it.)

> > Similarly, an R2L line will be continued like this:
> >
> >        +-------------------------------+
> >        |/RCSED-GNOL-YREV-YREV-YREV-YREV|
> >        |                         NOITPI|
> >        +-------------------------------+
> >
> > and truncated like this:
> >
> >        +-------------------------------+
> >        |$RCSED-GNOL-YREV-YREV-YREV-YREV|
> >        +-------------------------------+
> 
> This is the correct way.

The truncation and continuation will always be ``correct'' in L2R
lines with only left to right text, and in R2L lines with only
right-to-left text.  The ``problem'' happens only in mixed lines, and
then when the line is truncated or continued in the middle of a run of
text whose direction is the opposite of the line direction (actually,
of the base paragraph direction).

> The logical text is "some latin text followed by HEBREW VERY1 VERY2
> LONG TEXT FOR DEMONSTRATING OF THE DIFFERENT SCROLLING OPTIONS AND
> THE UNNATURAL RESULT OF ELIS APPROACH WITH small latin tail"
> 
> With your approach it will appear as (first, wider screen):
> 
>        +----------------------------------------+
>        |some latin text followed by HTIW HCAORP$|
>        |PA SILE FO TLUSER LARUTANNU EHT DNA SNO$|
>        |ITPO GNILLORCS TNEREFFID EHT FO GNITART$|
>        |SNOMED ROF TXET GNOL 2YREV 1YREV WERBEH$|
>        | small latin tail                       |
>        +----------------------------------------+
> 
> The Hebrew starts at 4th visual lines and goes upward.

No, this is some kind of misunderstanding, because there can be no
such display, not with the truncation glyphs.  Is the logical text
just a single line?  If so, the display will be like this:

       +----------------------------------------+
       |some latin text followed by HTIW HCAORP\|
       |PA SILE FO TLUSER LARUTANNU EHT DNA SNO\|
       |ITPO GNILLORCS TNEREFFID EHT FO GNITART\|
       |SNOMED ROF TXET GNOL 2YREV 1YREV WERBEH\|
       | small latin tail                       |
       +----------------------------------------+

And yes, the R2L text reads bottom to top.  But rewriting the central
piece of the display engine to make this use-case look better is
beyond my resources.  There's a lot of more important (IMO) issues to
take care of.  If this will really annoy users (and we won't know
until bidi Emacs hits the FTP sites, because these features are unique
to Emacs), someone else will have to come and do the surgery it takes,
sorry.

> with smaller screen (assume only 7 lines long):
> 
>        +--------------------+
>        |some latin text fol$|
>        |lowed by HTIW HCAOR$|
>        |PPA SILE FO TLUSER $|
>        |LARUTANNU EHT DNA S$|
>        |NOITPO GNILLORCS TN$|
>        |EREFFID EHT FO GNIT$|
>        |ARTSNOMED ROF TXET $|
>        +--------------------+

Again, I think you meant \ for continuation lines, not $ for
truncation.

> Oops, the beginning of the Hebrew text disappeared.

And this is significantly worse than if the _end_ of the text
disappears, because...?

>        +--------------------+
>        |$ 2YREV 1YREV WERBEH|
>        |$NOMED ROF TXET GNOL|
>        |$FID EHT FO GNITARTS|
>        |$PO GNILLORCS TNEREF|
>        |$TANNU EHT DNA SNOIT|
>        |$SILE FO TLUSER LARU|
>        |$???? HTIW HCAORPPA |
>        +--------------------+
> The last line ???? are "tail" by Eli's way and "smal" by my way.

Yes, but that continuation glyph should tell me and you that there's
more text out of view.

> I claim the scrolling should follow the "same" rules.
> e.g. 1: wide screen scrolled 20 (half width):
>        +----------------------------------------+
>        |$owd by OF TXET GNOL 2YREV 1YREV WERBEH$|
>        +----------------------------------------+

That's not the same rules applied to scrolling, that's exactly the
same situation of truncating a mixed L2R/R2L text as you described
above.

The issue with scrolling was when you have _two_ different paragraphs,
one with left-to-right base direction, the other with right-to-left.
Like this:

       +----------------------------------------+
       |some latin text                         |
       |                                        |
       |                        TXET WERBEH EMOS|
       +----------------------------------------+

(The empty line in between causes the last line to start a new
paragraph.)

The issue I raised was what should happen if I now type "C-u 2 C-x <".
My conclusion from this discussion (and thanks to all who contributed)
is that the resulting display should be this:

       +----------------------------------------+
       |$e latin text                           |
       |                                        |
       |                          TXET WERBEH E$|
       +----------------------------------------+

That is, each line scrolls towards its logical-order beginning.
"Logically-left", so to say.

(Hmm, does it mean "COPYLEFT" will be displayed as "THGIRYPOC"?
That's _really_ bad, isn't it?)

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[Richard Stallman]

    > What you describe here is the Emacs screen as rectangle frame moving
    > over the visual ordered text.

    For truncation and continuation, yes.  For horizontal scrolling, no:
    it works in the logical order.  That is, scroll-left moves text in
    each line towards its respective margin: left margin for L2R lines,
    right margin for R2L.  More about this near the end of this message.

Please don't do that.  It is counterintuitive, and contradicts the
idea of scrolling left and right.  When we have a horizontal scroll
bar, that behavior will make no sense at all.

Re: [emacs-bidi] Mixed L2R and R2L paragraphs and horizontal scroll

[Stefan Monnier]

>> What you describe here is the Emacs screen as rectangle frame moving
>> over the visual ordered text.

>     For truncation and continuation, yes.  For horizontal scrolling, no:
>     it works in the logical order.  That is, scroll-left moves text in
>     each line towards its respective margin: left margin for L2R lines,
>     right margin for R2L.  More about this near the end of this message.

> Please don't do that.  It is counterintuitive, and contradicts the
> idea of scrolling left and right.  When we have a horizontal scroll
> bar, that behavior will make no sense at all.

I agree that "inverted scrolling" will lead to surprising scrolling
behavior and "rigid scrolling" would lead to something more logical (at
least to my mind).  But I think it's "OK for now".


        Stefan

Re: [emacs-bidi] Mixed L2R and R2L paragraphs and horizontal scroll

[David Kastrup]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> What you describe here is the Emacs screen as rectangle frame moving
>>> over the visual ordered text.
>
>>     For truncation and continuation, yes.  For horizontal scrolling, no:
>>     it works in the logical order.  That is, scroll-left moves text in
>>     each line towards its respective margin: left margin for L2R lines,
>>     right margin for R2L.  More about this near the end of this message.
>
>> Please don't do that.  It is counterintuitive, and contradicts the
>> idea of scrolling left and right.  When we have a horizontal scroll
>> bar, that behavior will make no sense at all.
>
> I agree that "inverted scrolling" will lead to surprising scrolling
> behavior and "rigid scrolling" would lead to something more logical (at
> least to my mind).  But I think it's "OK for now".

Please note also that the keybindings C-x < and C-x > use mnemonics
based on graphical direction.

-- 
David Kastrup

Re: [emacs-bidi] Mixed L2R and R2L paragraphs and horizontal scroll

[Davis Herring]

> Please note also that the keybindings C-x < and C-x > use mnemonics
> based on graphical direction.

I can never remember whether the metaphor is sliding the text under a
fixed viewport or scanning a viewport over a fixed sheet of text, so that
the mnemonics would stop making sense doesn't make any difference to me.
;)

Davis

-- 
This product is sold by volume, not by mass.  If it appears too dense or
too sparse, it is because mass-energy conversion has occurred during
shipping.

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Thu, 04 Feb 2010 16:57:01 +0100
> Cc: emacs-bidi@gnu.org
> 
> Please note also that the keybindings C-x < and C-x > use mnemonics
> based on graphical direction.

We could invert the meaning in a buffer that forces R2L direction on
all its paragraphs.  We could also invert the meaning if point is on
an R2L line.  But the problem what to do with the lines of the other
directionality still needs a solution.  If someone can suggest a
better one, I'm all ears.  After all, that's why I started this
thread.

Re: [emacs-bidi] Mixed L2R and R2L paragraphs and horizontal scroll

[tomas]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Thu, Feb 04, 2010 at 04:57:01PM +0100, David Kastrup wrote:

[...]

> Please note also that the keybindings C-x < and C-x > use mnemonics
> based on graphical direction.

...which might be reversed under R2L affine keyboard layouts (but I
don't know for sure: there are better qualified people here to comment
on this)

regards
- -- tomás
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFLazIfBcgs9XrR2kYRApS3AJ4nZbTobxfAKZg7Kub2R2kswJCb+wCfdG8+
bo39ZdaQSkHO2J8/FAerJJU=
=juu3
-----END PGP SIGNATURE-----

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[Eli Zaretskii]

> Date: Thu, 4 Feb 2010 21:46:23 +0100
> From: tomas@tuxteam.de
> Cc: emacs-bidi@gnu.org, emacs-devel@gnu.org
> 
> > Please note also that the keybindings C-x < and C-x > use mnemonics
> > based on graphical direction.
> 
> ...which might be reversed under R2L affine keyboard layouts

That's system-dependent, AFAIU.  And having an R2L buffer current does
not mean your keyboard is set to an R2L language, anyway.

Re: [emacs-bidi] Mixed L2R and R2L paragraphs and horizontal scroll

[tomas]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Fri, Feb 05, 2010 at 12:23:19AM +0200, Eli Zaretskii wrote:

[...]

> > > Please note also that the keybindings C-x < and C-x > use mnemonics
> > > based on graphical direction.
> > 
> > ...which might be reversed under R2L affine keyboard layouts
> 
> That's system-dependent, AFAIU.  And having an R2L buffer current does
> not mean your keyboard is set to an R2L language, anyway.

Thanks for the insight :-)

My point was rather that one shouldn't take "mnemonicity" of keys for
granted. And then, what would the symbols "<" and ">" mean to a Hebrew
or Arabic writer anyway?

(Off on a tangent: this is another big difficulty internationalization
of Emacs will face: keybindings which make sense to an English speaking
american sitting in front of an US-Keyboard do sometimes make very
little sense on other keyboards (or worse: they are downright painful to
type)).

Regards
- -- tomás
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFLbQ8rBcgs9XrR2kYRAsA5AJ9JavD6S7WF73AWBzRzvcakawBpRACggdxC
wlhl28h8v9Cg3L9w5QJD0kc=
=zLcU
-----END PGP SIGNATURE-----

Re: [emacs-bidi] Mixed L2R and R2L paragraphs and horizontal scroll

[Stefan Monnier]

>> I agree that "inverted scrolling" will lead to surprising scrolling
>> behavior and "rigid scrolling" would lead to something more logical (at
>> least to my mind).  But I think it's "OK for now".
> Please note also that the keybindings C-x < and C-x > use mnemonics
> based on graphical direction.

That's indeed part of what is meant by "surprising scrolling behavior".


        Stefan

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> CC: ehud@unix.mvs.co.il, emacs-bidi@gnu.org, emacs-devel@gnu.org
> Date: Thu, 04 Feb 2010 06:01:38 -0500
> 
>     > What you describe here is the Emacs screen as rectangle frame moving
>     > over the visual ordered text.
> 
>     For truncation and continuation, yes.  For horizontal scrolling, no:
>     it works in the logical order.  That is, scroll-left moves text in
>     each line towards its respective margin: left margin for L2R lines,
>     right margin for R2L.  More about this near the end of this message.
> 
> Please don't do that.  It is counterintuitive, and contradicts the
> idea of scrolling left and right.

??? What are the alternatives?  The only one that was raised is the
``rigid scrolling'', but even there we need to do something sensible
for the nil value of ``line length''.

What are your practical suggestions?

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[Richard Stallman]

    ??? What are the alternatives?  The only one that was raised is the
    ``rigid scrolling'', but even there we need to do something sensible
    for the nil value of ``line length''.

Rigid scrolling is the right thing to do.  As far as I can see, it is
simple and well defined in all cases, including the case where the
window width controls the normal alignment of R2L lines.

That case (which is the one you have implemented so far, right?)  is a
matter of how to display the text when there is no hscroll.  However,
what hscroll should do in that case is totally straightforward: just
scroll all the text (including the parts that are hidden off the left
and right edges) rigidly left or right.

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Date: Fri, 05 Feb 2010 07:44:46 -0500
> Cc: ehud@unix.mvs.co.il, emacs-bidi@gnu.org, emacs-devel@gnu.org
> 
>     ??? What are the alternatives?  The only one that was raised is the
>     ``rigid scrolling'', but even there we need to do something sensible
>     for the nil value of ``line length''.
> 
> Rigid scrolling is the right thing to do.  As far as I can see, it is
> simple and well defined in all cases, including the case where the
> window width controls the normal alignment of R2L lines.
> 
> That case (which is the one you have implemented so far, right?)  is a
> matter of how to display the text when there is no hscroll.  However,
> what hscroll should do in that case is totally straightforward: just
> scroll all the text (including the parts that are hidden off the left
> and right edges) rigidly left or right.

You didn't say what to do when the (to be introduced) ``line length''
variable is nil.

Re: [emacs-bidi] Mixed L2R and R2L paragraphs and horizontal scroll

[Stefan Monnier]

> You didn't say what to do when the (to be introduced) ``line length''
> variable is nil.

Wouldn't that be window-width?


        Stefan

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: rms@gnu.org, ehud@unix.mvs.co.il, emacs-bidi@gnu.org, emacs-devel@gnu.org
> Date: Fri, 05 Feb 2010 13:06:49 -0500
> 
> > You didn't say what to do when the (to be introduced) ``line length''
> > variable is nil.
> 
> Wouldn't that be window-width?

Maybe, but I'm not sure.  Richard, could you please elaborate on your
suggestion?

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[Richard Stallman]

    > > You didn't say what to do when the (to be introduced) ``line length''
    > > variable is nil.
    > 
    > Wouldn't that be window-width?

    Maybe, but I'm not sure.  Richard, could you please elaborate on your
    suggestion?

I am not entirely sure what the issue is.

Perhaps it is my suggestion to create a variable to control where the
right margin should be for R2L lines.  I don't remember all the
details, but if it's an integer, that value should be the column at
which the beginning of an R2L line appears.  When it is nil, use the
window width (same as now).

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> CC: monnier@IRO.UMontreal.CA, emacs-bidi@gnu.org, ehud@unix.mvs.co.il,
> 	emacs-devel@gnu.org
> Date: Sat, 06 Feb 2010 17:55:25 -0500
> 
>     > > You didn't say what to do when the (to be introduced) ``line length''
>     > > variable is nil.
>     > 
>     > Wouldn't that be window-width?
> 
>     Maybe, but I'm not sure.  Richard, could you please elaborate on your
>     suggestion?
> 
> I am not entirely sure what the issue is.
> 
> Perhaps it is my suggestion to create a variable to control where the
> right margin should be for R2L lines.  I don't remember all the
> details, but if it's an integer, that value should be the column at
> which the beginning of an R2L line appears.  When it is nil, use the
> window width (same as now).

This means we need to allow scroll either left or right past the point
where the first character is at the window's edge.  That is,
window-hscroll will no longer return only zero or positive values.  Is
that reasonable?

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

> This means we need to allow scroll either left or right past the point
> where the first character is at the window's edge.  That is,
> window-hscroll will no longer return only zero or positive values.  Is
> that reasonable?

I think it is more reasonable than the alternatives.

It won't make horizontal scrollbar behavior in the presence of overfull
R-L and L-R lines all too great ("home" position being no longer either
fully left or right) but if you compare to the horizontal scrollbar
behavior when using inverted scrolling, it would be a haven of sanity.

-- 
David Kastrup

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[Richard Stallman]

    This means we need to allow scroll either left or right past the point
    where the first character is at the window's edge.  That is,
    window-hscroll will no longer return only zero or positive values.  Is
    that reasonable?

I think it is reasonable and necessary, once we have R2L text.  With
only L2R text, we have no need for a negative value of hscroll,
because scrolling the text away from the left window edge is useless.
With R2L text, that operation is useful because it can extend past
the left window edge.

Re: [emacs-bidi] Mixed L2R and R2L paragraphs and horizontal scroll

[Ehud Karni]

On Wed, 03 Feb 2010 20:59:13 Eli Zaretskii wrote:
>
> Maybe you are talking about how lines are re-flowed by these word
> processors.  If so, the Emacs equivalent is fill-paragraph, or the
> various packages (like longlines) that do it automatically for you.
> When a line is filled, a newline is inserted, and then the order of
> the text spread between several lines is what you expect:

OK, I want it to work like fill-paragraph, but I don't want to REALLY
do fill-paragraph, because I don't want to modify the file AT ALL.


>        +----------------------------------------+
>        |some latin text followed by HTIW HCAORP\|
>        |PA SILE FO TLUSER LARUTANNU EHT DNA SNO\|
>        |ITPO GNILLORCS TNEREFFID EHT FO GNITART\|
>        |SNOMED ROF TXET GNOL 2YREV 1YREV WERBEH\|
>        | small latin tail                       |
>        +----------------------------------------+
>
> And yes, the R2L text reads bottom to top.  But rewriting the central
> piece of the display engine to make this use-case look better is
> beyond my resources.  There's a lot of more important (IMO) issues to
> take care of.  If this will really annoy users (and we won't know
> until bidi Emacs hits the FTP sites, because these features are unique
> to Emacs), someone else will have to come and do the surgery it takes,
> sorry.

From experience (with my visual Hebrew package) this is very annoying.
Think of 3 very long (150, 270 and 150 characters) viewed on 80
characters wide screen, It is very confusing to find the start of
each line and to read upward 1 or 2 lines and go down again.


> > with smaller screen (assume only 7 lines long):
> >
> >        +--------------------+
> >        |some latin text fol$|
> >        |lowed by HTIW HCAOR$|
> >        |PPA SILE FO TLUSER $|
> >        |LARUTANNU EHT DNA S$|
> >        |NOITPO GNILLORCS TN$|
> >        |EREFFID EHT FO GNIT$|
> >        |ARTSNOMED ROF TXET $|
> >        +--------------------+
>
> Again, I think you meant \ for continuation lines, not $ for
> truncation.

Yes.

> > Oops, the beginning of the Hebrew text disappeared.
>
> And this is significantly worse than if the _end_ of the text
> disappears, because...?

Yes, it is definitely worse. First the the case of text that goes just
a screen and half down. You'll have to scroll one page down, read a
few lines upward, that scroll up, find the last line you read and
continue upward, you may need scroll up once more for the beginning of
the text. Second, consider the case of a very, very long text (more
than 100 continuation lines), it is almost impossible to find its
start with your way, while in my way it is just the normal way of
scrolling down (I do it by half screen at a time).


> That's not the same rules applied to scrolling, that's exactly the
> same situation of truncating a mixed L2R/R2L text as you described
> above.
>
> The issue with scrolling was when you have _two_ different paragraphs,
> one with left-to-right base direction, the other with right-to-left.

I don't think this issue is problematic per se. I think that if the
solution for R2L scrolling is good (when all the paragraphs are R2L)
then it can be applied even when combined with L2R paragraphs on the
same screen without any annoyance to the user.

Ehud.


--
 Ehud Karni           Tel: +972-3-7966-561  /"\
 Mivtach - Simon      Fax: +972-3-7976-561  \ /  ASCII Ribbon Campaign
 Insurance agencies   (USA) voice mail and   X   Against   HTML   Mail
 http://www.mvs.co.il  FAX:  1-815-5509341  / \
 GnuPG: 98EA398D <http://www.keyserver.net/>    Better Safe Than Sorry

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[Eli Zaretskii]

> Date: Thu, 4 Feb 2010 16:02:58 +0200
> From: "Ehud Karni" <ehud@unix.mvs.co.il>
> Cc: emacs-bidi@gnu.org, emacs-devel@gnu.org
> 
> On Wed, 03 Feb 2010 20:59:13 Eli Zaretskii wrote:
> >
> > Maybe you are talking about how lines are re-flowed by these word
> > processors.  If so, the Emacs equivalent is fill-paragraph, or the
> > various packages (like longlines) that do it automatically for you.
> > When a line is filled, a newline is inserted, and then the order of
> > the text spread between several lines is what you expect:
> 
> OK, I want it to work like fill-paragraph, but I don't want to REALLY
> do fill-paragraph, because I don't want to modify the file AT ALL.

Right, and I want $5 million on my bank account, but it doesn't
happen ;-)

Look, as you know, the bidirectional display reorders on the buffer
text level, not on the glyph level.  This means that in this example:

      name2 1234 catag2 NOITPIRCSED-GNOL-YREV name3

when the display engine, marching from left to right, reaches the
beginning of the R2L text (the character N), and calls the bidi
reordering code, that code runs all the way to just before "name3",
then starts delivering the characters backwards, from right to left.
This ``run to the end of R2L text'' happens below the level that
produces glyphs for display.  Therefore, the code which does this run
has no idea where the visible part of the line ends.  If it bumps into
a newline, it stops and goes backward, which is why filling a
paragraph displays like you expect.  But to do what you want in a
truncated or continued line means I need to throw away the entire
design and implementation of the bidi reordering.  I think it's too
heavy price to pay for a relatively rare situation (mixed L2R and R2L
text that hits the window margin at the wrong place).

> From experience (with my visual Hebrew package) this is very annoying.
> Think of 3 very long (150, 270 and 150 characters) viewed on 80
> characters wide screen, It is very confusing to find the start of
> each line and to read upward 1 or 2 lines and go down again.

Again, _only_ if these long lines mix R2L and L2R, and then _only_ if
the text that crosses the 80-column border has its direction against
the base paragraph direction.

> > > Oops, the beginning of the Hebrew text disappeared.
> >
> > And this is significantly worse than if the _end_ of the text
> > disappears, because...?
> 
> Yes, it is definitely worse. First the the case of text that goes just
> a screen and half down. You'll have to scroll one page down, read a
> few lines upward, that scroll up, find the last line you read and
> continue upward, you may need scroll up once more for the beginning of
> the text. Second, consider the case of a very, very long text (more
> than 100 continuation lines), it is almost impossible to find its
> start with your way, while in my way it is just the normal way of
> scrolling down (I do it by half screen at a time).

Maybe you forgot that we have line-move-visual since Emacs 23.1.
Anyway, I agree that for any limitation it's possible to find a
use-case, however marginal, where that limitation will be very
annoying.  (Yes, I think that 100 continuation lines, all of them
having R2L text when the paragraph is L2R, is a rare case.)

> I don't think this issue is problematic per se. I think that if the
> solution for R2L scrolling is good (when all the paragraphs are R2L)
> then it can be applied even when combined with L2R paragraphs on the
> same screen without any annoyance to the user.

Thanks.  Then help me convince Richard ;-)

Re: [emacs-bidi] Mixed L2R and R2L paragraphs and horizontal scroll

[Eli Zaretskii]

> Date: Thu, 04 Feb 2010 21:30:10 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-bidi@gnu.org, emacs-devel@gnu.org
> 
> Look, as you know, the bidirectional display reorders on the buffer
> text level, not on the glyph level.  This means that in this example:
> 
>       name2 1234 catag2 NOITPIRCSED-GNOL-YREV name3
> 
> when the display engine, marching from left to right, reaches the
> beginning of the R2L text (the character N), and calls the bidi
                             ^^^^^^^^^^^^^^^
Sorry, I meant the character V, of course, because the logical order
of this text is

      name2 1234 catag2 VERY-LONG-DESCRIPTION name3

Re: [emacs-bidi] Mixed L2R and R2L paragraphs and horizontal scroll

[tomas]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Thu, Feb 04, 2010 at 04:02:58PM +0200, Ehud Karni wrote:

[...]
> OK, I want it to work like fill-paragraph, but I don't want to REALLY
> do fill-paragraph, because I don't want to modify the file AT ALL. 

[Eli Zaretskii]

> >        +----------------------------------------+
> >        |some latin text followed by HTIW HCAORP\|
> >        |PA SILE FO TLUSER LARUTANNU EHT DNA SNO\|
> >        |ITPO GNILLORCS TNEREFFID EHT FO GNITART\|
> >        |SNOMED ROF TXET GNOL 2YREV 1YREV WERBEH\|
> >        | small latin tail                       |
> >        +----------------------------------------+
> >
> > And yes, the R2L text reads bottom to top.  But rewriting the central
> > piece of the display engine to make this use-case look better is
> > beyond my resources [...]

[Ehud Karni]

> >From experience (with my visual Hebrew package) this is very annoying.
> Think of 3 very long (150, 270 and 150 characters) viewed on 80
> characters wide screen, It is very confusing to find the start of
> each line and to read upward 1 or 2 lines and go down again.

Again, at first blush you are right, Ehud. It would be annoying. On the
other hand, wrapping a long line is just a display glitch, and the
display proposed by Eli is just more consistent.

I think what you want is visual-line-mode, and yes, this one would have
to tackle this job in the presence of R2L (i don't dare to think too
closely about *that*, my head starts to spin agein :)

Regards
- -- tomás
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFLbRG6Bcgs9XrR2kYRAiSkAJ9xrPbfHCdcPAKZPBQ7Vjx49ty/YwCfWk8F
MSTiQR/TuL6glq0+uOiCeFY=
=hlkf
-----END PGP SIGNATURE-----

Re: [emacs-bidi] Mixed L2R and R2L paragraphs and horizontal scroll

[Davis Herring]

> What you describe here is the Emacs screen as rectangle frame moving
> over the visual ordered text. This is technically sound but very wrong
> from a user view point. The reason is that Hebrew reader see the
> "continuation" lines in reverse order, and has to read this from the
> last (continuation) line (the most down one) upward.

The difference between a L2R line and an R2L line is not in the visual
order of characters in them (that's determined by the directionality of
the text itself, of course), but in the layout of pieces of text.  The
question is, what is a "piece of text"?  Certainly the (logical) boundary
between L2R and R2L text is also the boundary of such pieces.  Consider
this rendering of a L2R line:

 +------------------------------+
 |the ordinals TSRIF and DNOCES |
 +------------------------------+

The two R2L words appear to be in the wrong order if you read only them or
if you interpret the line as R2L, in which case it's the rendering of the
logical line "SECOND and FIRST the ordinals".

Suppose we narrow the window:

 +-----------------+
 |the ordinals ???\|
 |?? and DNOCES    |
 +-----------------+

If we interpret the ??? and ?? as two different pieces of R2L text, we
have split the logical string "FIRST" into "FIR" and "ST", and so we
render

 +-----------------+
 |the ordinals RIF\|
 |TS and DNOCES    |
 +-----------------+

with the two pieces presented in their logical order (the piece that
occurs earlier in the buffer is on an earlier screen line).  If, however,
we consider the ????? as one piece, presented in two places merely as a
rearrangement of glyphs on the screen, we render

 +-----------------+
 |the ordinals TSR\|
 |IF and DNOCES    |
 +-----------------+

Neither of these is perfect: in the first case, in addition to the line
break there are also two breaks (logically in the middle of "RS") that
disrupt the flow of the text more than the usual break between L2R and
R2L.  In the second case, as you say, the continuation runs bottom-to-top,
which is undesirable.

However, both of them are usable.  In the first case, the reader applies
this algorithm upon encountering a R2L character after a L2R stretch:

 1. Scan rightward until an L2R character or the right end of the screen
line; remember which was encountered.
 2. Read R2L from here to the starting position.
 3. If the end of the screen line was encountered, seek to the left end of
the next screen line and note it as the new starting position; go to 1.

In the second case, the algorithm is

 1. Scan rightward until an L2R character (and go to step 3) or the right
end of the screen line.
 2. Seek to the left end of the next screen line, and go to 1.
 3. Read R2L, bottom-to-top from here to the starting position.

I don't claim to know which technique R2L readers would prefer; I am not
one.  The first has the disadvantage that you must interleave scanning for
the (visual) end of the R2L string, while the second has the disadvantage
that you must read bottom-to-top.

I tend to prefer the second interpretation, partly because no characters
on the first screen line move or change when the window width changes
(unless they are removed entirely by narrowing).  Also because the process
to find the point from which to read is the same as the process involved
in reading the surrounding L2R text anyway.  Also because it makes the
continuation lines a rigid rearrangement of pieces of the longer screen
line we'd have in the ideal case.  (This last point is a question of "what
do we mean continuation lines to mean as currently implemented, anyway?",
so there's no automatic right answer.)

> No word processor that uses that approach (check open-office with a
> long enough text, or with the other OS word processor).

Word processors don't have continuation lines that are meant to be
interpreted as one long screen line.

>> What this means is that an L2R line that ends with a stretch of R2L
>> text will be continued as follows:
>>
>>        +-------------------------------+
>>        |name2 1234 catag2 NOITPIRCSED-\|
>>        |GNOL-YREV                      |
>>        +-------------------------------+
>
> See the problem - the user must start reading from the second line.

What is the alternative?  (What you proposed later in your message doesn't
address this case to my understanding.)  Perhaps you would want this?

 +-------------------------------+
 |name2 1234 catag2 ED-GNOL-YREV\|
 |                     NOITPIRCS |
 +-------------------------------+

Of course, we would need some sort of separator/indicator in the (visual)
"2 E" space to indicate that it was the middle and not the end of the R2L
text.  And it would be really odd to have

 +---------------------------------------+
 |name2 1234 catag2 OITPIRCSED-GNOL-YREV\|
 |               with some latin after N |
 +---------------------------------------+

become

 +----------------------------------------+
 |name2 1234 catag2 NOITPIRCSED-GNOL-YREV |
 |with some latin after                   |
 +----------------------------------------+

when the window is widened by one column.

>> and truncated thusly:
>>
>>        +-------------------------------+
>>        |name2 1234 catag2 NOITPIRCSED-$|
>>        +-------------------------------+
>
> The truncation is technically correct but wrong from user perspective.

This is how word processors truncate when they do.  In that mode they show
a (rectangular) subset of the visual layout of the overall document.

> I claim the scrolling should follow the "same" rules.
> e.g. 1: wide screen scrolled 20 (half width):
>        +----------------------------------------+
>        |$owd by OF TXET GNOL 2YREV 1YREV WERBEH$|
>        +----------------------------------------+
> e.g. 2: wide screen scrolled 60 (1.5 width):
>        +----------------------------------------+
>        |$LLORCS TNEREFFID EHT FO GNITARTSNOMED $|
>        +----------------------------------------+
> e.g. 3: wide screen scrolled 150:
>        +----------------------------------------+
>        |$HTIW H small latin tail                |
>        +----------------------------------------+
>
> The word "same" is between quotes because it is the same appearance
> even though the rules may be slightly different.

It looks odd to me to have two pieces of text on the same line move in
different directions when you scroll the window.

New sub-topic: what does one do with the (logical) text

 he said, "SHE SAID, 'latin again.' TODAY." yesterday.

?  Stripping the punctuation for simplicity, it would seem that it would
get rendered as

 he said DIAS EHS latin again YADOT yesterday

which seems confusing because the L2R text brackets its quotation but the
R2L text doesn't.  I bring this up in this thread because, if the right
answer is to render it as

 he said YADOT latin again DIAS EHS yesterday

(with some sort of punctuation or special graphical indication as to how
far the reader must seek to the right before beginning the R2L scan), then
it may have bearing on the present scrolling discussion.

Davis

-- 
This product is sold by volume, not by mass.  If it appears too dense or
too sparse, it is because mass-energy conversion has occurred during
shipping.

Re: Bidirectional embeddings (was: Mixed L2R and R2L paragraphs and horizontal scroll)

[Eli Zaretskii]

> Date: Wed, 3 Feb 2010 13:02:28 -0800 (PST)
> From: "Davis Herring" <herring@lanl.gov>
> Cc: eliz@gnu.org, emacs-bidi@gnu.org, emacs-devel@gnu.org
> 
> New sub-topic:

New topic asks for a new subject.

> what does one do with the (logical) text
> 
>  he said, "SHE SAID, 'latin again.' TODAY." yesterday.
> 
> ?  Stripping the punctuation for simplicity, it would seem that it would
> get rendered as
> 
>  he said DIAS EHS latin again YADOT yesterday
> 
> which seems confusing because the L2R text brackets its quotation but the
> R2L text doesn't.  I bring this up in this thread because, if the right
> answer is to render it as
> 
>  he said YADOT latin again DIAS EHS yesterday
> 
> (with some sort of punctuation or special graphical indication as to how
> far the reader must seek to the right before beginning the R2L scan), then
> it may have bearing on the present scrolling discussion.

You are talking about embedding text, i.e. there's a part of this line
that should be rendered as if it belonged to an R2L line, although the
line is L2R.

The solution to this is already described in UAX#9: use the special
RLE or RLO formatting characters to start the embedding and the PDF
character to end it.

Without these formatting characters, the rendering will be as in your
first example.

I think this has nothing to do with the original topic of scrolling,
continuation and truncation.

Re: [emacs-bidi] Mixed L2R and R2L paragraphs and horizontal scroll

[Ehud Karni]

On Wed, 3 Feb 2010 13:02:28 Davis Herring wrote:
> >
> > See the problem - the user must start reading from the second line.
>
> What is the alternative?  (What you proposed later in your message doesn't
> address this case to my understanding.)  Perhaps you would want this?
>
>  +-------------------------------+
>  |name2 1234 catag2 ED-GNOL-YREV\|
>  |                     NOITPIRCS |
>  +-------------------------------+

No. I want this:
   +-------------------------------+
   |name2 1234 catag2 ED-GNOL-YREV\|
   |NOITPIRCS                      |
   +-------------------------------+

And again, the same in R2L direction:
   +-------------------------------+
   |ED-GNOL-YREV name2 1234 catag2\|
   |                      NOITPIRCS|
   +-------------------------------+

And the logical text "VERY-LONG-DESCRIPTION name2 1234 catag2"
would be rendered (R2L paragraph):
   +-------------------------------+
   |/name2 12 NOITPIRCSED-GNOL-YREV|
   |                      34 catag2|
   +-------------------------------+

This how it is done in Hebrew newspapers and in HTML, and this is how
Hebrew readers are used to it (of course newspapers and HTML don't
breaks words, but that is the only difference).

e.g. http://www.cancer.org.il/template_site2/default.asp?textSearch=&maincat=15&catid=368&pageid=3331&innerparentId=4361
look for "nutrition and physical" and see how the English title is
broken into 2 lines and is right justified.

I wish more users who uses Hebrew routinely will take part in this
discussion.

Ehud.


--
 Ehud Karni           Tel: +972-3-7966-561  /"\
 Mivtach - Simon      Fax: +972-3-7976-561  \ /  ASCII Ribbon Campaign
 Insurance agencies   (USA) voice mail and   X   Against   HTML   Mail
 http://www.mvs.co.il  FAX:  1-815-5509341  / \
 GnuPG: 98EA398D <http://www.keyserver.net/>    Better Safe Than Sorry

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[Eli Zaretskii]

> Date: Thu, 4 Feb 2010 18:21:06 +0200
> From: "Ehud Karni" <ehud@unix.mvs.co.il>
> Cc: eliz@gnu.org, emacs-bidi@gnu.org, emacs-devel@gnu.org
> 
> And the logical text "VERY-LONG-DESCRIPTION name2 1234 catag2"
> would be rendered (R2L paragraph):
>    +-------------------------------+
>    |/name2 12 NOITPIRCSED-GNOL-YREV|
>    |                      34 catag2|
>    +-------------------------------+
> 
> This how it is done in Hebrew newspapers and in HTML, and this is how
> Hebrew readers are used to it (of course newspapers and HTML don't
> breaks words, but that is the only difference).

Please!  Newspapers don't have truncated and continued lines, they
have newlines between every two lines.  With newlines, the bidi
display will show exactly what you (and every other Hebrew reader)
expect.

> e.g. http://www.cancer.org.il/template_site2/default.asp?textSearch=&maincat=15&catid=368&pageid=3331&innerparentId=4361
> look for "nutrition and physical" and see how the English title is
> broken into 2 lines and is right justified.

That's exactly what I see in bidi Emacs, when there's a newline
between "cancer" and "treatment".  THIS IS NOT THE USE-CASE WE ARE
DISCUSSING IN THIS THREAD.  That use-case works like you want.

> I wish more users who uses Hebrew routinely will take part in this
> discussion.

And I wish they'd also step forward to help me coding.

Re: [emacs-bidi] Mixed L2R and R2L paragraphs and horizontal scroll

[Stefan Monnier]

>> This how it is done in Hebrew newspapers and in HTML, and this is how
>> Hebrew readers are used to it (of course newspapers and HTML don't
>> breaks words, but that is the only difference).
> Please!  Newspapers don't have truncated and continued lines, they
> have newlines between every two lines.  With newlines, the bidi
> display will show exactly what you (and every other Hebrew reader)
> expect.

I'm not sure whether newpapers really have such newlines (at least for
paper newspapers, it's a philosophical question).  But I think Ehud's
argument makes sense if you consider that currently (i.e. in L2R only
text), adding a newline where Emacs wraps a line, leads to "the same
display" (modulo line-wrapping glyphs in the fringe and things like
that, obviously).  And I understand Ehud's argument as saying that the
same should hold for R2L and mixed text.

It seems at least a reasonable point of view.  Is your point of view
based on the fact that it seems difficult/impossible to implement, or is
it based on the fact that you think Ehud's expectation is weird?

If it's just "difficult", then (just like rigid scrolling), it can be
kept as a known shortcoming.

How do other line-wrapping text editors with bidi support behave with
such long mixed-L2R-R2L lines?


        Stefan

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: ehud@unix.mvs.co.il, emacs-bidi@gnu.org, emacs-devel@gnu.org
> Date: Thu, 04 Feb 2010 17:13:29 -0500
> 
> >> This how it is done in Hebrew newspapers and in HTML, and this is how
> >> Hebrew readers are used to it (of course newspapers and HTML don't
> >> breaks words, but that is the only difference).
> > Please!  Newspapers don't have truncated and continued lines, they
> > have newlines between every two lines.  With newlines, the bidi
> > display will show exactly what you (and every other Hebrew reader)
> > expect.
> 
> I'm not sure whether newpapers really have such newlines (at least for
> paper newspapers, it's a philosophical question).  But I think Ehud's
> argument makes sense if you consider that currently (i.e. in L2R only
> text), adding a newline where Emacs wraps a line, leads to "the same
> display" (modulo line-wrapping glyphs in the fringe and things like
> that, obviously).  And I understand Ehud's argument as saying that the
> same should hold for R2L and mixed text.
> 
> It seems at least a reasonable point of view.  Is your point of view
> based on the fact that it seems difficult/impossible to implement, or is
> it based on the fact that you think Ehud's expectation is weird?

I'm sorry I didn't make myself clear enough on that, to the degree
that such a question should at all be asked.

Like Ehud, I think that it would be swell to have what he wants.  But,
possibly unlike Ehud, I think that what I have now it not a disaster,
and we can live with it for the time being, maybe even longer.

The reasons for my decision to implement truncation and continuation
as I did are:

  . It is the only reasonable way to go that does not call for a very
    serious surgery, perhaps even a total rewrite, of the display
    engine code.

  . I saw no other editor that supports truncation and behaves
    otherwise.  (I don't know about any editors that support
    continuation lines like Emacs does.)  See below.

  . The issue pops up only in relatively rare situations: mixed
    L2R/R2L text that gets truncated/continued within a stretch of
    text whose directionality is against the paragraph direction.

> If it's just "difficult", then (just like rigid scrolling), it can be
> kept as a known shortcoming.

It is either VERY difficult or very slow.

The current display code lays out glyphs in each ``glyph row'' one by
one, in the visual order.  Thus, for the portion of text that is
reversed from its logical order, the bidi reordering code effectively
delivers the characters backwards to this glyph layout code, in the
decreasing order of buffer positions.  That is, the glyphs assembled
first are the last ones to be read.  Then you hit the window margin,
and know that there isn't enough place for the whole line.  Only then
you know how many characters will fit on this line.  But you know that
in terms of the last portion of the text in the reading order, which
tells you very little about how many characters at the beginning of
this stretch of text you could display instead.  (Remember that Emacs
supports variable size characters and different fonts on the same
line, so just counting characters will not do.)

What would be nice is to scan the text to be reversed in the logical
order, and find the part of it that will fit on this screen line.
Then we could reorder only that part.  But to do that, we need to try
every possibility by actually doing most of the display work behind
the scenes, because of the complications with different font sizes,
faces, composite characters and issues like ligatures and the like,
which change the amount of screen estate taken by a portion of a line,
even if you just juxtapose the same two characters.

With a newline marking the end of the line, it's easy: the bidi
reordering ends at the newline, then restarts after it.  By contrast,
to support ``bidi-smart continuation'', we need to find the place
where to break the line, and that is impossible without actually
trying to display it.

In the example below

  word1 word2 WORD1 WORD2

to be displayed as 

  word1 word2 2DROW 1DROW

if the window is only wide enough to display

  word1 word2  1DROW

we need to try displaying in order

  word1 word2 1
  word1 word2 1D
  word1 word2 1DR
  word1 word2 1DRO
  word1 word2 1DROW
  word1 word2  1DROW
  word1 word2 W 1DROW

until we discover where we should stop.  (We could do a binary search,
of course, but that's details.)  I don't think that's reasonable, and
I have no idea what will this do to the redisplay speed.

> How do other line-wrapping text editors with bidi support behave with
> such long mixed-L2R-R2L lines?

MS Word truncates in the visual order, like I suggest.  It doesn't
have continuation lines (it re-flows instead, which is the same as our
fill-paragraph, and that works like Ehud wants already.)  I didn't
have a chance of checking OpenOffice; I can do that next week, if no
one beats me to it.

I have Yudit installed, but I cannot get it to display Hebrew text (it
shows hex numbers instead); if someone knows how to do that on
Windows, please tell.  But someone told in this thread that it, too,
re-flows.  If that's true, we have no prior art that's different from
what I have now in the bidi Emacs.

Of course, Emacs breaks ground where many other similar tools punt...

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[Eli Zaretskii]

> Date: Fri, 05 Feb 2010 11:50:58 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-bidi@gnu.org, emacs-devel@gnu.org
> 
> I have Yudit installed, but I cannot get it to display Hebrew text (it
> shows hex numbers instead); if someone knows how to do that on
> Windows, please tell.  But someone told in this thread that it, too,
> re-flows.  If that's true, we have no prior art that's different from
> what I have now in the bidi Emacs.

This last part seems to be incorrect: Yudit does display continuation
lines like Ehud wants (as much as I could conclude by looking at the
UTF-8 code points it displays instead of characters).  So we do have
``prior art'' here.

Re: [emacs-bidi] Mixed L2R and R2L paragraphs and horizontal scroll

[Stefan Monnier]

>> It seems at least a reasonable point of view.  Is your point of view
>> based on the fact that it seems difficult/impossible to implement, or is
>> it based on the fact that you think Ehud's expectation is weird?

> I'm sorry I didn't make myself clear enough on that, to the degree
> that such a question should at all be asked.

> Like Ehud, I think that it would be swell to have what he wants.  But,
> possibly unlike Ehud, I think that what I have now it not a disaster,
> and we can live with it for the time being, maybe even longer.

That's what I gathered, I just wanted it to be clear.
BTW, That's perfectly fine by me, in case it matters ;-)


        Stefan

Re: [emacs-bidi] Mixed L2R and R2L paragraphs and horizontal scroll

[David De La Harpe Golden]

[-- Attachment #1: Type: text/plain, Size: 2228 bytes --]

Eli Zaretskii wrote:

> 
> I have Yudit installed, but I cannot get it to display Hebrew text (it
> shows hex numbers instead); if someone knows how to do that on
> Windows, please tell. 

Well, I don't really, but some searching turned up

http://indictrans.in/en/documentation/howto/yuditwindowshowto

Which suggests you have to copy a ttf (presumably with the relevant 
glyphs...) to the fonts directory under your yudit installation, rather 
than expecting it to pick up windows installed fonts,  and declare it 
ttf in the yudit.properties file.

There's also
http://www.yudit.org/en/howto/configure/
and question 8 in the FAQ looks relevant:
http://www.yudit.org/en/faq/

I don't really know yudit well, I just installed it via the debian 
package manager, it apparently looked after creating a proper 
yudit.properties file for me (attached, though obviously for debian).

> But someone told in this thread that it, too,
> re-flows.

N.B. I only looked at whole lines of R2L or L2R in it previously - note 
in the previous screenshot that any newlines in the yudit buffer are 
shown as visible blue "enter" glyphs, otherwise you can assume they're 
continuation lines. i.e. I didn't mean re-flow as in re-fill (inserting 
newline bytes into the file as necessary), I meant re-flow as in wrap 
into continuation lines. Possibly that was a bad use of re-flow on my 
part, I dunno.  (In context of the previous post in question, it was to 
point out that yudit doesn't _do_ truncation, only wrapping (and uses 
the window margins for wrap column), so the question of horizontal 
scrolling doesn't arise in it.)

I did not previously look at single lines of mixed R2L and L2R text 
being wrapped into continuation lines.   As far as I can tell, an L2R 
line will be continued at the left even if wrapped at an R2L span 
within it.  An R2L line will be continued at the right even if wrapping 
at a L2R span within it.

I've attached new screenshots, before and after resizing the window to 
cause wrapping, anyway.  yudit has word boundary wrapping, so there's 
two after shots, one with it on and one with it off (see FAQ Q18). 
They're probably not all that helpful compared to having a working
yudit to play with though.


[-- Attachment #2: yuditbefore.png --]
[-- Type: image/png, Size: 44621 bytes --]

[-- Attachment #3: yuditafter_wordwrapoff.png --]
[-- Type: image/png, Size: 48297 bytes --]

[-- Attachment #4: yuditafter_wordwrapon.png --]
[-- Type: image/png, Size: 48143 bytes --]

[-- Attachment #5: yudit.properties.gz --]
[-- Type: application/x-gzip, Size: 4296 bytes --]

Re: [emacs-bidi] Mixed L2R and R2L paragraphs and horizontal scroll

[Ehud Karni]

On Sat, 06 Feb 2010 13:39:11 David De La Harpe Golden wrote:
>
> I did not previously look at single lines of mixed R2L and L2R text
> being wrapped into continuation lines.   As far as I can tell, an L2R
> line will be continued at the left even if wrapped at an R2L span
> within it.  An R2L line will be continued at the right even if wrapping
> at a L2R span within it.

That is the Hebrew readers are used to it (i.e. in HTML pages and
printed text - newspapers and articles).

Eli has explained the difficulties to do it with the current Emacs
display engine. Emacs has some properties (text overlays is a major
example) that make it even more difficult. Until a better bidi
display engine is in place, I'll write my own elisp code to do the
re-flow on long mixed lines, so the actual work will be acceptable.


--
 Ehud Karni           Tel: +972-3-7966-561  /"\
 Mivtach - Simon      Fax: +972-3-7976-561  \ /  ASCII Ribbon Campaign
 Insurance agencies   (USA) voice mail and   X   Against   HTML   Mail
 http://www.mvs.co.il  FAX:  1-815-5509341  / \
 GnuPG: 98EA398D <http://www.keyserver.net/>    Better Safe Than Sorry

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[Eli Zaretskii]

> Date: Sat, 06 Feb 2010 13:39:11 +0000
> From: David De La Harpe Golden <david@harpegolden.net>
> CC: Stefan Monnier <monnier@IRO.UMontreal.CA>, ehud@unix.mvs.co.il, 
>  emacs-bidi@gnu.org, emacs-devel@gnu.org
> 
> > I have Yudit installed, but I cannot get it to display Hebrew text (it
> > shows hex numbers instead); if someone knows how to do that on
> > Windows, please tell. 
> 
> Well, I don't really, but some searching turned up
> 
> http://indictrans.in/en/documentation/howto/yuditwindowshowto
> 
> Which suggests you have to copy a ttf (presumably with the relevant 
> glyphs...) to the fonts directory under your yudit installation, rather 
> than expecting it to pick up windows installed fonts,  and declare it 
> ttf in the yudit.properties file.
> 
> There's also
> http://www.yudit.org/en/howto/configure/
> and question 8 in the FAQ looks relevant:
> http://www.yudit.org/en/faq/

I've read all of these, and even found and fixed some problems they
don't mention (e.g., the yudit.properties file sets data and font
paths to something Unixy, like /usr/share/yudit or
/home/gsinai/build/share/yudit).  It still shows only hex numbers
instead of text in R2L languages.

A Windows port that does not automatically use the installed fonts
is not really a port, IMO.  Strangely, I didn't find anything on the
Internet that would help; evidently, there aren't too many Windows
users of Yudit.

Re: [emacs-bidi] Mixed L2R and R2L paragraphs and horizontal scroll

[David De La Harpe Golden]

Eli Zaretskii wrote:

> A Windows port that does not automatically use the installed fonts
> is not really a port, IMO. 

Not that it helps, but it itself doesn't really automatically do so on 
gnu+linux either - the debian packagers have just glossed over the issue 
by having the package install a config pointing at some system font 
directories.

Actually, turns out gnome gedit has bidi support too and seems to handle 
the wrapping much like yudit, and there are apparently windows binaries 
available these days - might be worth a shot if you're determined to 
stick to windows:

http://live.gnome.org/Gedit/Windows
http://ftp.gnome.org/pub/gnome/binaries/win32/gedit/2.29/

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[Beni Cherniavsky]

On Thu, Feb 4, 2010 at 18:21, Ehud Karni <ehud@unix.mvs.co.il> wrote:
> I wish more users who uses Hebrew routinely will take part in this
> discussion.
>
That'd be my clue ;-).  Hi.  New here, sufficiently retro-lurked.

[Sorry, long mail.  In the first half I'm whining about why I don't
like Eli's solution; but I also reply with technical ideas below...]


First, I want to draw attention to the distinction between line
wrapping and truncation/scrolling.

- Line wrapping (aka continuation lines) in visual order is bad!
  It violates the deep axiom that reading order between *lines*
  is downward regardless of bidi.

- Truncation in visual order is OK!  It fits the rigid scrolling
  model of a small window horizontally moving over a wide page.

  This is the only model of truncation that I've seen in other
  programs.  You either have the "Web" model where lines wrap at
  window boundary and there is no truncation, or the "Page" model
  where lines are layed out onto an underlying page, and you see
  a physical window onto it.

  (It does require a lot of horizontal scrolling to read mixed
  direction text, but that's the user's problem.)

- Truncation in logical order might(?) be OK if coupled with
  logical-order "mirrored" scrolling.

  I've never seen such a program, so I don't know if it would be
  usable.  I believe we can easily try it out by running a plain
  L2R emacs in a bidi terminal, e.g. mlterm.  I'll try to work with
  that a bit to see how it feels...

In all the following, I'm only talking about wrapped continuation
lines.  I got the impression Ehud is also mostly concerned about
contintinuation lines - correct me if I'm wrong.


Second, allow me to sum up Ehud's arguments.

- It's the Right Thing to do.  Books, papers, and correct software have
  always done it this way, and that's what the Unicode standard says.

- Convenience: Doing it the wrong way requires discontinued reading,
  which is annoying.

and add 2 more angles to the issue:

- Mental model, or why imperfect bidi is painful:

  As an R2L user, I constantly maintain a mental model of the logical
  order.  I've got some deep habits and assumptions about the mapping
  from logical to visual.  *Any* deviation will completely confuse my
  poor brain about the logical order of the buffer.

  Worse yet, if I now proceed to *edit* the buffer, I'll modify it in
  completely wrong places, and even when I realize that, fixing it will
  be even harder!  I'll need to *simultaneously* reverse-engineer your
  deviant bidi algorithm and figure out the real logical order, and
  then very carefully fix my edits, all the time getting strangely
  permuted feedback for my actions.

  This involves concentration, a lot of forward/backward-char movement
  to visualize the logical order, and expletives under my breath :-(

  This is the *real* reason we hate broken bidi support.  No bidi at all
  is frequently better - ain't pretty but at least has 1:1 mental model.

- Emotional: this kind of broken bottom-up line wrapping is precisely
  the problem with visual-order Hebrew.  Reduce the browser window width
  on any visual-order site, and you'll see it.  We (Hebrew readers)
  had to live with it until logical-order support arrived in browsers,
  have cursed too many sites that use visual-order to this very day,
  and by now we hate it with a burning passion!

  It's none of your fault, but getting line wrapping wrong will step on
  very sore spots with many users...

To be fair, we're talking about rare situations where embedded text is
broken across lines.  But note that a wrong base direction can inflict
this on whole paragraphs (more on that below).


On Fri, Feb 5, 2010 at 11:50, Eli Zaretskii <eliz@gnu.org> wrote:
> Like Ehud, I think that it would be swell to have what he wants.  But,
> possibly unlike Ehud, I think that what I have now it not a disaster,
> and we can live with it for the time being, maybe even longer.
>
> The reasons for my decision to implement truncation and continuation
> as I did are:
>
>  . It is the only reasonable way to go that does not call for a very
>    serious surgery, perhaps even a total rewrite, of the display
>    engine code.
>
>  . I saw no other editor that supports truncation and behaves
>    otherwise.  (I don't know about any editors that support
>    continuation lines like Emacs does.)  See below.
>
Truncation is OK, but the issue is continuation.

Not following your claim about editors that support continuation -
all these do and behave otherwise (i.e. as Ehud wants):
Notepad, gedit, firefox/webkit, OpenOffice.

>  . The issue pops up only in relatively rare situations: mixed
>    L2R/R2L text that gets truncated/continued within a stretch of
>    text whose directionality is against the paragraph direction.
>
Indeed, embedded text tends to be short.

But I'm afraid it's bigger than you think, because if the base direction
of a paragraph is incorrect, *the whole paragraph* will wrap in this
broken bottom-up manner.  Since base direction guessing is never perfect,
and users don't always have the option - or patience - to fix it manually,
this makes the otherwise minor problem more visible.

Also, changing the base direction of any paragraph will behave funny:
Instead of (mostly) just jumping horizontally, it'll also reverse the
order of lines!

                                                            !of lines
Instead of (mostly) just jumping horizontally, it'll also reverse the
:Also, changing the base direction of any paragraph will behave funny

See?  [estimated, some punctuation might be off]

This also means that forcing all paragraphs to R2L or L2R base direction
(which would be a handy way to momentarily work around wrong imperfect
guessing) would break line order in half the paragraphs in a mixed buffer!


>> If it's just "difficult", then (just like rigid scrolling), it can be
>> kept as a known shortcoming.
>
> It is either VERY difficult or very slow.
>
> The current display code lays out glyphs in each ``glyph row'' one by
> one, in the visual order.  Thus, for the portion of text that is
> reversed from its logical order, the bidi reordering code effectively
> delivers the characters backwards to this glyph layout code, in the
> decreasing order of buffer positions.  That is, the glyphs assembled
> first are the last ones to be read.  Then you hit the window margin,
> and know that there isn't enough place for the whole line.  Only then
> you know how many characters will fit on this line.  But you know that
> in terms of the last portion of the text in the reading order, which
> tells you very little about how many characters at the beginning of
> this stretch of text you could display instead.  (Remember that Emacs
> supports variable size characters and different fonts on the same
> line, so just counting characters will not do.)
>
> What would be nice is to scan the text to be reversed in the logical
> order, and find the part of it that will fit on this screen line.
> Then we could reorder only that part.

Right.  Line breaking must be done in logical order.

> But to do that, we need to try
> every possibility by actually doing most of the display work behind
> the scenes, because of the complications with different font sizes,
> faces, composite characters and issues like ligatures and the like,
> which change the amount of screen estate taken by a portion of a line,
> even if you just juxtapose the same two characters.
>
Right, this is a known annoying property of bidi interacting with
typographic features.  Note however that you have a new trade-off here:
if you could compromise precision of line breaking to get correct bidi
behaviour (with fast redisplay), users would be happy.

See below for a concrete attempt.

> With a newline marking the end of the line, it's easy: the bidi
> reordering ends at the newline, then restarts after it.

So if only the line breaking points were static, you'd have no
performance problem!

=> Could you maybe cache this information and recompute it only when
the line is edited?  I understand part of the whole point of your
implementation was to avoid any caching of bidi ordering; but caching
of line breaking points sounds much less intrusive...

[XEmacs already has a "Line Start Cache" according to its Internals Manual.
I didn't find a similar overview for Emacs.  Is there anything I can read
to understand Emacs redisplay before I attempt to approach the source?]

> By contrast,
> to support ``bidi-smart continuation'', we need to find the place
> where to break the line, and that is impossible without actually
> trying to display it.
>
> In the example below
>
>  word1 word2 WORD1 WORD2
>
> to be displayed as
>
>  word1 word2 2DROW 1DROW
>
> if the window is only wide enough to display
>
>  word1 word2  1DROW
>
> we need to try displaying in order
>
>  word1 word2 1
>  word1 word2 1D
>  word1 word2 1DR
>  word1 word2 1DRO
>  word1 word2 1DROW
>  word1 word2  1DROW
>  word1 word2 W 1DROW
>
> until we discover where we should stop.  (We could do a binary search,
> of course, but that's details.)  I don't think that's reasonable, and
> I have no idea what will this do to the redisplay speed.
>
Binary search is a big improvement!  In 10 attempts you can handle lines
of 1K chars, in 20 - 1M.  On my computer Emacs presently handles 100k
smoothly, 1M already feels sluggish.  By crude (and probably wrong)
computation, binary search would still be fast enough up to 10K...

Also, I presume that the heavy part of a redisplay is normally the actual
output to screen (if not, why do such a complex job minimizing it?).  This
means that "dry" running the engine without actual output 10 times should
result in much less that 10x slowdown.

To top this, I think you can do several times better if you allow some
imprecision in line breaking of mixed-direction paragraphs.  Naturally,
you must not overshoot the screen, but some undershooting is OK.  So it
seems to me that you could reasonably do it with a greedy approach:

(1) Add characters in *logical order* as long as they fit.
(2) Try it in visual order to account for precise typographic stuff.
(3) As long as it doesn't fit, strip one a char and retry (2).
(4) When OK, repeat with actual output display to the screen.

If (1) overestimates, you're left with a shorter line than ideal; if
it underestimates, you do extra iterations.  But I guess that it
normally won't be off by more than one character, so it will look OK
and run fast.

[One pathological case that springs to mind is Arabic shaping.  Doing
(1) in logical order would result in all the wrong ligatures, risking
the estimation being seriously off.  It's still much better than wrong
line order, so I'd ignore that for now; but an Arabic expert opinion
would be welcome...]

Note that this scheme runs the display engine at least 3 times, even
for pure-L2R short lines!  We'd have to optimize the common cases before
a release; I can see how it might work, but I don't want to complicate
the picture at this stage.

As long as we conclude that SOME such scheme is workable, we can
leave the detailed implementation for the future.


Finally, I want to propose a feature that I think will be handy,
and also happens to support efficient wrapping.  The truth is that any
way to wrap an embedding accross lines is ugly!  I'd like a mode where
any embedding either fits completely on a line or starts and ends on a
lines by itself:

+----------------------------------------+
|some latin text followed by            \|
|\          ROF TXET GNOL TAHWEMOS WERBEH|
|\                     SIHT GNITARTSNOMED|
|followed by latin tail                  |
+----------------------------------------+

This is relatively easy to implement efficiently - you add embedded
characters in *visual* order as you propose, but if the embedding
doesn't fit entirely, you just fall back to the breaking where the
embedding started!  You don't even need a stack - I'm talking one
"primary" level for each visual line.

If you don't like any of the other ideas, this seems like a minimally
intrusive way to make your approach more usable.

-- 
Beni Cherniavsky-Paskin <cben@users.sf.net>

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[Eli Zaretskii]

> From: Beni Cherniavsky <cben@users.sf.net>
> Date: Thu, 11 Feb 2010 23:40:03 +0200
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, emacs-bidi@gnu.org, emacs-devel@gnu.org
> 
> [Sorry, long mail.  In the first half I'm whining about why I don't
> like Eli's solution; but I also reply with technical ideas below...]

Thank you for your comments.

> - Truncation in logical order might(?) be OK if coupled with
>   logical-order "mirrored" scrolling.

What is ``logical-order "mirrored" scrolling''?

>   Worse yet, if I now proceed to *edit* the buffer, I'll modify it in
>   completely wrong places, and even when I realize that, fixing it will
>   be even harder!  I'll need to *simultaneously* reverse-engineer your
>   deviant bidi algorithm and figure out the real logical order, and
>   then very carefully fix my edits, all the time getting strangely
>   permuted feedback for my actions.

I don't think fixing such problems is anywhere near that hard.  Just
display the text in its logical order (a flip of a buffer-local
variable) and fix it.  Case closed.

That doesn't mean that we should proliferate problems that need
fixing, of course.

>   This is the *real* reason we hate broken bidi support.

This is not really fair.  I'm not developing a ``broken bidi
support''.  Everything developed so far is first-class bidirectional
operation, as much as I know what that means, at least on the low
level on which I'm working.  Line continuation is the first issue
where I gave up on making it 100% perfect, because it's just too damn
hard for someone who has only weekends to work on that, and doesn't
have enough knowledge and experience in Emacs display-related
features.

The truth is that this is not a job for a bidi expert, it is a job for
an Emacs display engine expert who can ask for bidi advice from time
to time.

But I've waited for 8 years for such a display engine expert to come
and integrate the bidi reordering code I wrote with Emacs redisplay.
It never happened.  So now I'm doing it as best I can, in the hope
that my best will be good enough.

People who want the result to be better can help by suggesting very
specific implementation ideas, based on specific details of the Emacs
redisplay code.  General ideas are generally not helpful, because my
problem is not with principles, it is with the details.  So if you
want to help, please make yourself familiar with xdisp.c, which is the
bulk of the display engine.  Then if you have specific ideas about how
to implement continuation lines that read in the correct order, I'll
be all ears.

>   No bidi at all
>   is frequently better - ain't pretty but at least has 1:1 mental model.

Maybe I should just quit, then.  I don't want my name scribbled on
code that is considered worse than what Emacs was before that.  I
could find better uses for my scarce free time.

> To be fair, we're talking about rare situations where embedded text is
> broken across lines.  But note that a wrong base direction can inflict
> this on whole paragraphs (more on that below).

Unlike with other programs, in Emacs, wrong base direction is very
easy to fix, even with what I have now.  There's a per-buffer variable
that forces one of the two directions on all the paragraphs in the
buffer, and if that's not fine-grained enough, you can insert the
corresponding directional mark at the beginning of the paragraph.
(Eventually, there should be commands to do this, without asking the
user to remember the Unicode codepoints of these characters.)

> >  . I saw no other editor that supports truncation and behaves
> >    otherwise.  (I don't know about any editors that support
> >    continuation lines like Emacs does.)  See below.
> >
> Truncation is OK, but the issue is continuation.
> 
> Not following your claim about editors that support continuation -
> all these do and behave otherwise (i.e. as Ehud wants):
> Notepad, gedit, firefox/webkit, OpenOffice.

No, they don't have continuation lines.  They reflow the lines,
i.e. divide text differently between the lines.

> Indeed, embedded text tends to be short.
> 
> But I'm afraid it's bigger than you think, because if the base direction
> of a paragraph is incorrect, *the whole paragraph* will wrap in this
> broken bottom-up manner.

See above: this is easy to fix in Emacs.  Again, that's not a good
reason to have incorrect display, of course.

> This also means that forcing all paragraphs to R2L or L2R base direction
> (which would be a handy way to momentarily work around wrong imperfect
> guessing) would break line order in half the paragraphs in a mixed buffer!

Let's not get carried away: the problem will only be with continued
lines, not with every line.

> So if only the line breaking points were static, you'd have no
> performance problem!

If the line breaking points were at known buffer positions, yes.  (I
don't quite know what you mean by ``static''.)

> => Could you maybe cache this information and recompute it only when
> the line is edited?

This is unlikely to help, with the current design of the display
engine.  It works very hard to avoid redisplaying lines that did not
change.  So when it finally decides to redisplay a line, there are
very good chances that the cached value is invalid anyway.  Note that
even placing a text property on some text of a line could make it
overflow the display margin at a different point: for example, giving
the text bold or italics face is all you need for the continuation
point to move.

> [XEmacs already has a "Line Start Cache" according to its Internals Manual.

So does Emacs.  But the cache is invalidated when/where the text
changes.

> I didn't find a similar overview for Emacs.  Is there anything I can read
> to understand Emacs redisplay before I attempt to approach the source?]

Only the comments at the beginning of xdisp.c.  After that, read the
code, guided by the high-level structure described in those comments.
That, asking questions here, and related discussions on this list is
all you have, sorry.

> > until we discover where we should stop.  (We could do a binary search,
> > of course, but that's details.)  I don't think that's reasonable, and
> > I have no idea what will this do to the redisplay speed.
> >
> Binary search is a big improvement!  In 10 attempts you can handle lines
> of 1K chars, in 20 - 1M.  On my computer Emacs presently handles 100k
> smoothly, 1M already feels sluggish.  By crude (and probably wrong)
> computation, binary search would still be fast enough up to 10K...

The problem is not with implementing binary search, the problem is to
plug it into the display engine, which needs to be aware of all the
possible side effects of each thing it does while preparing display of
a line.

> Also, I presume that the heavy part of a redisplay is normally the actual
> output to screen (if not, why do such a complex job minimizing it?).

I don't think so.  I think the heavy part is computing and merging
faces.  But doing measurements to find out the hot spots would be in
itself a useful project, which may help down the line.  Volunteers are
welcome.

> To top this, I think you can do several times better if you allow some
> imprecision in line breaking of mixed-direction paragraphs.  Naturally,
> you must not overshoot the screen, but some undershooting is OK.  So it
> seems to me that you could reasonably do it with a greedy approach:
> 
> (1) Add characters in *logical order* as long as they fit.
> (2) Try it in visual order to account for precise typographic stuff.
> (3) As long as it doesn't fit, strip one a char and retry (2).
> (4) When OK, repeat with actual output display to the screen.

I think this will only work if there are no embeddings.  In text with
embeddings, doing it in logical order will have you scan wrong
characters anyway, and you are back at square one.

> Finally, I want to propose a feature that I think will be handy,
> and also happens to support efficient wrapping.  The truth is that any
> way to wrap an embedding accross lines is ugly!  I'd like a mode where
> any embedding either fits completely on a line or starts and ends on a
> lines by itself:
> 
> +----------------------------------------+
> |some latin text followed by            \|
> |\          ROF TXET GNOL TAHWEMOS WERBEH|
> |\                     SIHT GNITARTSNOMED|
> |followed by latin tail                  |
> +----------------------------------------+

Again, what about embeddings?  I want to have full UAX#9 support, not
just some simplified bidi that breaks as soon as the user wants to use
the full power of the Unicode algorithm.

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[Eli Zaretskii]

> Date: Fri, 12 Feb 2010 13:03:52 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-bidi@gnu.org, emacs-devel@gnu.org
> 
> > So if only the line breaking points were static, you'd have no
> > performance problem!
> 
> If the line breaking points were at known buffer positions, yes.

Actually, that's not true, in general.  Embeddings break that.

Here's an example.  Imagine the following text in logical order:

  he said "I HEARD HIM SAYING ['car is CAR in arabic'] AND FAINTED"

The [...] part is an embedding: it is there to have this text display
as

  he said "DETNIAF DNA 'car is RAC in arabic' GNIYAS MIH DRAEH I"

rather than as

  he said "GNIYAS MIH DRAEH I 'car is RAC in arabic' DETNIAF DNA"

The latter display does not express what the writer of the text wanted
to say.  (Please don't nit-pick wrt the known issues with reordering
quotes and apostrophes: they are here just to make the text more
readable, and can be easily removed without affecting the fundamental
problem I'm talking about.)  UAX#9 allows up to 60 levels of such
embeddings, btw.  The characters [ and ] represent special Unicode
code points known as LRE and PDF, respectively.  There are also LRO,
RLE, and RLO, for similar jobs but with the other direction.

Now, suppose the window is only wide enough to display this:

  he said "'car is GNIYAS MIH DRAEH I"

The continuation line should then display the rest of the text:

  "DETNIAF DNA RAC in arabic'

(again, disregard the placement of quotes).  Right?  Or does this look
incorrect?  If so, what is the correct display in this case?

Anyway, if we stop the bidi reordering after "car is", and restart it
at "CAR", the information about the unfinished embedding is lost, and
the continuation line will be displayed differently.

Which means breaking the line in the middle of an embedding is tricky
at best.  It also means that putting a newline on some whitespace
character, like fill-paragraph does, will have the same problem of
generating incorrect (or at least unexpected) display of the second
part of the broken line.

Conclusion: to apply such smart line continuation, we need first to
establish where are the points at which we are allowed to break the
line.

Re: [emacs-bidi] Mixed L2R and R2L paragraphs and horizontal scroll

[Ehud Karni]

On Thu, 04 Feb 2010 21:40:32 Eli Zaretskii wrote:
>
> Please!  Newspapers don't have truncated and continued lines, they
> have newlines between every two lines.  With newlines, the bidi
> display will show exactly what you (and every other Hebrew reader)
> expect.

I think you got it wrong here. Newspaper article are written a whole
paragraph without new lines on a wide screen and then reflowed to
the column width. You can do it with any word processor, and change
the width when you print and get the same behavior.

> > e.g. http://www.cancer.org.il/template_site2/default.asp?textSearch=&maincat=15&catid=368&pageid=3331&innerparentId=4361
> > look for "nutrition and physical" and see how the English title is
> > broken into 2 lines and is right justified.

> That's exactly what I see in bidi Emacs, when there's a newline
> between "cancer" and "treatment".  THIS IS NOT THE USE-CASE WE ARE
> DISCUSSING IN THIS THREAD.  That use-case works like you want.

You keep saying "when there's a newline" and "use fill-paragraph"
because of your technical constraint (explained in another email).

I propose the following: use a virtual "right-margin-goal-column" (a
real one if given, and if not it is the screen width), do a virtual
`fill-paragraph' on the line, and do the bidi processing on the
result, BUT without inserting any real new-lines.

It has the shortcoming of not going exactly up to right edge (or left
edge for R2L paragraph), but it will have the benefit of not breaking
words and allowing normal reading for long line with embedding text of
the opposite direction.

It does not solve or even relate to the scrolling strategy.

Ehud.


--
 Ehud Karni           Tel: +972-3-7966-561  /"\
 Mivtach - Simon      Fax: +972-3-7976-561  \ /  ASCII Ribbon Campaign
 Insurance agencies   (USA) voice mail and   X   Against   HTML   Mail
 http://www.mvs.co.il  FAX:  1-815-5509341  / \
 GnuPG: 98EA398D <http://www.keyserver.net/>    Better Safe Than Sorry

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[Eli Zaretskii]

> Date: Fri, 5 Feb 2010 14:21:49 +0200
> From: "Ehud Karni" <ehud@unix.mvs.co.il>
> Cc: herring@lanl.gov, emacs-bidi@gnu.org, emacs-devel@gnu.org
> 
> On Thu, 04 Feb 2010 21:40:32 Eli Zaretskii wrote:
> >
> > Please!  Newspapers don't have truncated and continued lines, they
> > have newlines between every two lines.  With newlines, the bidi
> > display will show exactly what you (and every other Hebrew reader)
> > expect.
> 
> I think you got it wrong here. Newspaper article are written a whole
> paragraph without new lines on a wide screen and then reflowed to
> the column width. You can do it with any word processor, and change
> the width when you print and get the same behavior.

Emacs's way of re-flowing is by inserting newlines.  Continuation
lines are not it, because they break the line in an arbitrary place,
like the middle of a word.  This is a no-no in any newspaper article.

> I propose the following: use a virtual "right-margin-goal-column" (a
> real one if given, and if not it is the screen width), do a virtual
> `fill-paragraph' on the line, and do the bidi processing on the
> result, BUT without inserting any real new-lines.

I explained elsewhere why this does not solve the problem: the
reordering code works below the glyph layout code, so it has no idea
when it has reached the right-margin-goal-column.  At least I don't
know how to do that; ideas are welcome.

The reordering code can do what you want if we somehow tell it at
which _buffer_position_ to stop, as if it hits a newline or a
paragraph separator.  But buffer positions are not equivalent to
display positions, due to display features like:

  . variable fonts and faces
  . glyphs that come from overlays, `display' properties, images, etc.
  . `wrap-prefix' and `line-prefix' variables
  . composite characters
  . tab expansion

(probably forgot a few).  Note that the last one affects text
terminals as well, so even there you cannot simply count characters.

> It has the shortcoming of not going exactly up to right edge (or left
> edge for R2L paragraph), but it will have the benefit of not breaking
> words and allowing normal reading for long line with embedding text of
> the opposite direction.

The problem is that, for the reasons I explain above and in a previous
message in this thread, what you suggest will still end up overflowing
the margin sometimes, and we are back with the same problem.  (If I
understood your suggestion, that is; if not, please point out where
I'm wrong.)

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[Miles Bader]

Eli Zaretskii <eliz@gnu.org> writes:
> Emacs's way of re-flowing is by inserting newlines.

Hmm, that used to be the case, but what about the `word-wrap' variable
(etc)?

[That, incidentally, is another reason we need a "line-length" variable 
(I thought it was on a todo somewhere...)]

-Miles

-- 
((lambda (x) (list x x)) (lambda (x) (list x x)))

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[Eli Zaretskii]

> From: Miles Bader <miles@gnu.org>
> Cc: ehud@unix.mvs.co.il,  emacs-bidi@gnu.org,  emacs-devel@gnu.org
> Date: Fri, 05 Feb 2010 23:22:19 +0900
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> > Emacs's way of re-flowing is by inserting newlines.
> 
> Hmm, that used to be the case, but what about the `word-wrap' variable

Right.  So I have one argument less, but the rest are still valid.

> (etc)?

What's that?

> That, incidentally, is another reason we need a "line-length" variable 

Why is that related?  AFAICS, word-wrap does not affect horizontal
scrolling.

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[Miles Bader]

Eli Zaretskii <eliz@gnu.org> writes:
>> > Emacs's way of re-flowing is by inserting newlines.
>> 
>> Hmm, that used to be the case, but what about the `word-wrap' variable
>
> Right.  So I have one argument less, but the rest are still valid.
>
>> (etc)?
>
> What's that?

Oh, I was just referring to the ancillary stuff, like visual-line-mode.

>> That, incidentally, is another reason we need a "line-length" variable 
>
> Why is that related?  AFAICS, word-wrap does not affect horizontal
> scrolling.

word-wrapping also needs a way for the user to specify a
less-than-window-width wrapping-point.

I dunno if the two notions of "line-length" can/should be conflated,
just tossing out the notion.

-Miles

-- 
Marriage, n. The state or condition of a community consisting of a master, a
mistress and two slaves, making in all, two.

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[Eli Zaretskii]

> From: Miles Bader <miles@gnu.org>
> Cc: emacs-bidi@gnu.org,  emacs-devel@gnu.org
> Date: Sat, 06 Feb 2010 10:07:05 +0900
> 
> >> That, incidentally, is another reason we need a "line-length" variable 
> >
> > Why is that related?  AFAICS, word-wrap does not affect horizontal
> > scrolling.
> 
> word-wrapping also needs a way for the user to specify a
> less-than-window-width wrapping-point.

I thought this is always the window margin, isn't it?

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[Miles Bader]

Eli Zaretskii <eliz@gnu.org> writes:
>> word-wrapping also needs a way for the user to specify a
>> less-than-window-width wrapping-point.
>
> I thought this is always the window margin, isn't it?

It currently is -- the ability to specify a different wrap-point hasn't
been implemented yet.  It's a desired feature though.

[Perhaps most useful solution is something a bit more complex,
e.g. which lets the user set max/min line-lengths, and uses the
window-width for anything between those.  But anyway... not sure if it
has anything to do with RtoL.]

-Miles

-- 
Christian, n. One who follows the teachings of Christ so long as they are not
inconsistent with a life of sin.

Re: [emacs-bidi] Mixed L2R and R2L paragraphs and horizontal scroll

[Ehud Karni]

On Fri, 05 Feb 2010 15:47:37 Eli Zaretskii <eliz@gnu.org> wrote:
>
> The reordering code can do what you want if we somehow tell it at
> which _buffer_position_ to stop, as if it hits a newline or a
> paragraph separator.  But buffer positions are not equivalent to
> display positions, due to display features like:
>
>   . variable fonts and faces
>   . glyphs that come from overlays, `display' properties, images, etc.
>   . `wrap-prefix' and `line-prefix' variables
>   . composite characters
>   . tab expansion
>
> (probably forgot a few).  Note that the last one affects text
> terminals as well, so even there you cannot simply count characters.

I see your point. I don't think variable fonts, tabs or composite
characters are a real problem (tabs are taken into account in
fill-paragraph, and the others can be handled by setting the
`right-margin' to a value below the screen width). The text overlays
are a real problem when they exist.

> The problem is that, for the reasons I explain above and in a previous
> message in this thread, what you suggest will still end up overflowing
> the margin sometimes, and we are back with the same problem.  (If I
> understood your suggestion, that is; if not, please point out where
> I'm wrong.)

I understand, but since I'm using wide screens (130 columns at work,
over 180 at home), I can set the my line length for the virtual
`fill-paragraph' to a lower value (say 110) and hope that it won't
overflow, even if it is exceeded by some characters.

Ehud.


--
 Ehud Karni           Tel: +972-3-7966-561  /"\
 Mivtach - Simon      Fax: +972-3-7976-561  \ /  ASCII Ribbon Campaign
 Insurance agencies   (USA) voice mail and   X   Against   HTML   Mail
 http://www.mvs.co.il  FAX:  1-815-5509341  / \
 GnuPG: 98EA398D <http://www.keyserver.net/>    Better Safe Than Sorry

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[Eli Zaretskii]

> Date: Sat, 6 Feb 2010 17:42:36 +0200
> From: "Ehud Karni" <ehud@unix.mvs.co.il>
> Cc: emacs-bidi@gnu.org, emacs-devel@gnu.org
> 
> >   . variable fonts and faces
> >   . glyphs that come from overlays, `display' properties, images, etc.
> >   . `wrap-prefix' and `line-prefix' variables
> >   . composite characters
> >   . tab expansion
> >
> > (probably forgot a few).  Note that the last one affects text
> > terminals as well, so even there you cannot simply count characters.
> 
> I see your point. I don't think variable fonts, tabs or composite
> characters are a real problem (tabs are taken into account in
> fill-paragraph, and the others can be handled by setting the
> `right-margin' to a value below the screen width).

How much below?  There's no way of knowing that.  A font used by a
face can be arbitrarily large.

> > The problem is that, for the reasons I explain above and in a previous
> > message in this thread, what you suggest will still end up overflowing
> > the margin sometimes, and we are back with the same problem.  (If I
> > understood your suggestion, that is; if not, please point out where
> > I'm wrong.)
> 
> I understand, but since I'm using wide screens (130 columns at work,
> over 180 at home), I can set the my line length for the virtual
> `fill-paragraph' to a lower value (say 110) and hope that it won't
> overflow, even if it is exceeded by some characters.

Well, that's hardly a general solution.

Re: [emacs-bidi] Mixed L2R and R2L paragraphs and horizontal scroll

[Ehud Karni]

On Tue, 02 Feb 2010 22:04:51 Eli Zaretskii wrote:
> > >
> > > A real example will help here.
> >
> > Here is a real example (upper case = Hebrew)
> > PUT THE FILES IN
> > /dir1/sdr2/sdr3
> > AND I WILL TAKE THEM
> >
> > This will appear as (think of 160 character wide text screen):
> >                                                    NI SELIF EHT TUP
> > /dir1/sdr2/sdr3
> >                                                MEHT EKAT LLIW I DNA
>
> And why would ``rigid scrolling'' help in this case?

This has nothing to do with "rigid scrolling" (or I don't understand
this concept), and everything with `right-margin-goal-column' which
is a logical right edge of the screen for R2L display.

In a wide screen the above will be shown as:
    +------------------------------------------------------------+
    |                                            NI SELIF EHT TUP|
    |/dir1/sdr2/sdr3                                             |
    |                                        MEHT EKAT LLIW I DNA|
    +------------------------------------------------------------+

In a less wide screen it will be shown as:
    +----------------------------+
    |            NI SELIF EHT TUP|
    |/dir1/sdr2/sdr3             |
    |        MEHT EKAT LLIW I DNA|
    +----------------------------+

In a wide screen with `right-margin-goal-column` set to 30:
    +------------------------------------------------------------+
    |              NI SELIF EHT TUP                              |
    |/dir1/sdr2/sdr3                                             |
    |           EHT EKAT LLIW I DNA                              |
    +------------------------------------------------------------+

I hope this is clearer.

Ehud.


--
 Ehud Karni           Tel: +972-3-7966-561  /"\
 Mivtach - Simon      Fax: +972-3-7976-561  \ /  ASCII Ribbon Campaign
 Insurance agencies   (USA) voice mail and   X   Against   HTML   Mail
 http://www.mvs.co.il  FAX:  1-815-5509341  / \
 GnuPG: 98EA398D <http://www.keyserver.net/>    Better Safe Than Sorry

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[Eli Zaretskii]

> Date: Wed, 3 Feb 2010 15:22:19 +0200
> From: "Ehud Karni" <ehud@unix.mvs.co.il>
> Cc: emacs-bidi@gnu.org, emacs-devel@gnu.org
> 
> In a wide screen the above will be shown as:
>     +------------------------------------------------------------+
>     |                                            NI SELIF EHT TUP|
>     |/dir1/sdr2/sdr3                                             |
>     |                                        MEHT EKAT LLIW I DNA|
>     +------------------------------------------------------------+
> 
> In a less wide screen it will be shown as:
>     +----------------------------+
>     |            NI SELIF EHT TUP|
>     |/dir1/sdr2/sdr3             |
>     |        MEHT EKAT LLIW I DNA|
>     +----------------------------+
> 
> In a wide screen with `right-margin-goal-column` set to 30:
>     +------------------------------------------------------------+
>     |              NI SELIF EHT TUP                              |
>     |/dir1/sdr2/sdr3                                             |
>     |           EHT EKAT LLIW I DNA                              |
>     +------------------------------------------------------------+
> 
> I hope this is clearer.

Yes, it's clear.  But why is the last variant better than the others?
Is there really significance to the relative horizontal positions of
characters in different lines that makes it important to preserve that
relation?

Re: [emacs-bidi] Mixed L2R and R2L paragraphs and horizontal scroll

[Ehud Karni]

On Wed, 03 Feb 2010 21:01:17 Eli Zaretskii wrote:
> >
> > In a wide screen with `right-margin-goal-column` set to 30:
> >     +------------------------------------------------------------+
> >     |              NI SELIF EHT TUP                              |
> >     |/dir1/sdr2/sdr3                                             |
> >     |           EHT EKAT LLIW I DNA                              |
> >     +------------------------------------------------------------+
> >
> > I hope this is clearer.
>
> Yes, it's clear.  But why is the last variant better than the others?
> Is there really significance to the relative horizontal positions of
> characters in different lines that makes it important to preserve that
> relation?

Think of 160 characters wide screen with short lines. It is difficult
to follow the order of the lines when some appear at left side and
some are on the right side.

Ehud.


--
 Ehud Karni           Tel: +972-3-7966-561  /"\
 Mivtach - Simon      Fax: +972-3-7976-561  \ /  ASCII Ribbon Campaign
 Insurance agencies   (USA) voice mail and   X   Against   HTML   Mail
 http://www.mvs.co.il  FAX:  1-815-5509341  / \
 GnuPG: 98EA398D <http://www.keyserver.net/>    Better Safe Than Sorry

Re: Mixed L2R and R2L paragraphs and horizontal scroll

[Stefan Monnier]

> Here's a peculiar design decision that needs to be made for the bidi
> display: how to display horizontally scrolled lines in a buffer with
> mixed L2R and R2L paragraphs.

That's a tricky question indeed.  I don't know if there are other text
editors with similar problems from which to get inspiration.

So it seems we have 3 options:
- "saturated scrolling" where hscroll can be positive or negative and
  l2r lines won't scroll at all when hscroll is negative and vice-versa.
- "inverted scrolling" where hscroll is always positive and is
  interpreted as scrolling in the direction of the line, so the
  direction can change from one line to the next.
- "rigid scrolling" where the window-width determines the distance
  between l2r's leftmost position and r2l's rightmost position and then
  the text is taken as extending both left and right from those, so
  hscroll can be positive or negative and lines can be scrolled "before
  their beginning".

Inverted scrolling seem to be easier to implement.  It has the downside
that C-x > will scroll "the other way" in R2L contexts, but maybe it's
not that terrible: there's always a source of confusion when talking
about scrolling direction anyway, because we can either think of
scrolling the text under a fixed window, or scrolling a window over
a fixed text.


        Stefan