"... the window start at a meaningless point within a line."

"... the window start at a meaningless point within a line."

[Alan Mackenzie]

Hello, Emacs.

In the Elisp manual, page "Window Start and End", it's described how, if
in (set-window-start x) x is a position not at the start of a line, the
display engine will, instead of x, choose some value near x as the
window start.  Is this really necessary?  It causes problems.

The particular problem I'm looking at is with Follow Mode, where a pair
of adjacent windows are of unequal width.  This is a very common
scenario - I frequently use Follow Mode with three side by side windows
of widths 79, 79, and 80 (as measured by window-body-width).

The problem occurs when there's a long line at the bottom of window 2
and its continuation line at the top of window 3.  The display manager
chops off the line in W2 after 78 characters, and inserts a \ sign.  It
then starts the continuation line in W3 at character 80 (counting from
1), assuming as it does, that the previous line in this window would be
79 characters long.  So character 79 appears neither on the first
portion of the line in W2, nor in its continuation in W3.  This is
suboptimal.

This problem would not occur if there were a flag to set-window-start
meaning "this value really is where I want the window to start, not some
beginning of line nearby".

How difficult would it be to relax the constraint on window start
positions and implement this flag?

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: "... the window start at a meaningless point within a line."

[Eli Zaretskii]

> Date: Wed, 30 Sep 2015 20:45:13 +0000
> From: Alan Mackenzie <acm@muc.de>
> 
> In the Elisp manual, page "Window Start and End", it's described how, if
> in (set-window-start x) x is a position not at the start of a line, the
> display engine will, instead of x, choose some value near x as the
> window start.  Is this really necessary?

The reason for this is that we want the physical beginning of the line
(i.e. the character after a newline or at bob) always be aligned at
the left edge of the window's text area, even if it is currently out
of view.  This prevents annoying horizontal scrolling of long lines
when the beginning of the line comes into view.

> The particular problem I'm looking at is with Follow Mode, where a pair
> of adjacent windows are of unequal width.  This is a very common
> scenario - I frequently use Follow Mode with three side by side windows
> of widths 79, 79, and 80 (as measured by window-body-width).
> 
> The problem occurs when there's a long line at the bottom of window 2
> and its continuation line at the top of window 3.  The display manager
> chops off the line in W2 after 78 characters, and inserts a \ sign.  It
> then starts the continuation line in W3 at character 80 (counting from
> 1), assuming as it does, that the previous line in this window would be
> 79 characters long.  So character 79 appears neither on the first
> portion of the line in W2, nor in its continuation in W3.  This is
> suboptimal.
> 
> This problem would not occur if there were a flag to set-window-start
> meaning "this value really is where I want the window to start, not some
> beginning of line nearby".
> 
> How difficult would it be to relax the constraint on window start
> positions and implement this flag?

It's not difficult, at least in principle: the recalculation of the
window-start point is done in a single function.  Not sure if it will
have adverse effects, though.  You will surely have that horizontal
scrolling, but I don't know what other effects will be caused.

But let me turn the table and ask you: how difficult would it be to
make sure follow-mode never lets a window end in the middle of a
continued line?

Re: "... the window start at a meaningless point within a line."

[Alan Mackenzie]

Hello, Eli.

On Thu, Oct 01, 2015 at 11:48:40AM +0300, Eli Zaretskii wrote:
> > Date: Wed, 30 Sep 2015 20:45:13 +0000
> > From: Alan Mackenzie <acm@muc.de>

> > In the Elisp manual, page "Window Start and End", it's described how, if
> > in (set-window-start x) x is a position not at the start of a line, the
> > display engine will, instead of x, choose some value near x as the
> > window start.  Is this really necessary?

> The reason for this is that we want the physical beginning of the line
> (i.e. the character after a newline or at bob) always be aligned at
> the left edge of the window's text area, even if it is currently out
> of view.  This prevents annoying horizontal scrolling of long lines
> when the beginning of the line comes into view.

OK.  But we're really talking here about where continuation lines start,
not physical BOLs.

> > The particular problem I'm looking at is with Follow Mode, where a pair
> > of adjacent windows are of unequal width.  This is a very common
> > scenario - I frequently use Follow Mode with three side by side windows
> > of widths 79, 79, and 80 (as measured by window-body-width).

> > The problem occurs when there's a long line at the bottom of window 2
> > and its continuation line at the top of window 3.  The display manager
> > chops off the line in W2 after 78 characters, and inserts a \ sign.  It
> > then starts the continuation line in W3 at character 80 (counting from
> > 1), assuming as it does, that the previous line in this window would be
> > 79 characters long.  So character 79 appears neither on the first
> > portion of the line in W2, nor in its continuation in W3.  This is
> > suboptimal.

> > This problem would not occur if there were a flag to set-window-start
> > meaning "this value really is where I want the window to start, not some
> > beginning of line nearby".

> > How difficult would it be to relax the constraint on window start
> > positions and implement this flag?

> It's not difficult, at least in principle: the recalculation of the
> window-start point is done in a single function.  Not sure if it will
> have adverse effects, though.  You will surely have that horizontal
> scrolling, but I don't know what other effects will be caused.

I think scrolling such a window one screen line down would cause the
break in the line to have to be repositioned.  So it wouldn't be trivial.

> But let me turn the table and ask you: how difficult would it be to
> make sure follow-mode never lets a window end in the middle of a
> continued line?

In the worst case, very difficult.  Indeed, with three follow windows,
all of slightly different widths, and a fiendish specially constructed
file, it could be that when you scroll the buffer a single screen line to
deal with a break between W1 and W2, you create the same problem between
W2 and W3.  In such a scenario, you might end up scrolling the buffer
quite a long way in the search for no "broken" continued lines at either
boundary.  With such a file there might be NO position where there isn't
a break.

In real life?  I imagine it's quite common to use Follow Mode with very
narrow windows (say, four at ~60 wide) when the annoyance of continued
lines is less important than seeing all of a long function on the screen
at once.  There will then be lots of continued lines, and 3 window
boundaries, W1-W2, W2-W3, W3-W4.

If we were to go this route (of repositioning to avoid line breaks
between follow windows), there would have to be a limit on how far one
could scroll, with a value such as 3.  This might have to be a
customisation variable.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: "... the window start at a meaningless point within a line."

[Eli Zaretskii]

> Date: Thu, 1 Oct 2015 09:41:38 +0000
> Cc: emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > > In the Elisp manual, page "Window Start and End", it's described how, if
> > > in (set-window-start x) x is a position not at the start of a line, the
> > > display engine will, instead of x, choose some value near x as the
> > > window start.  Is this really necessary?
> 
> > The reason for this is that we want the physical beginning of the line
> > (i.e. the character after a newline or at bob) always be aligned at
> > the left edge of the window's text area, even if it is currently out
> > of view.  This prevents annoying horizontal scrolling of long lines
> > when the beginning of the line comes into view.
> 
> OK.  But we're really talking here about where continuation lines start,
> not physical BOLs.

The display engine always starts from the physical BOL when it lays
out buffer text.  That's because only at a physical BOL it knows that
the window-relative X coordinate is zero.  Otherwise, there's no
anchor for it to start at.

Invalidating this basic assumption will cause strange effects,
including the horizontal scrolling I mentioned.

> > But let me turn the table and ask you: how difficult would it be to
> > make sure follow-mode never lets a window end in the middle of a
> > continued line?
> 
> In the worst case, very difficult.  Indeed, with three follow windows,
> all of slightly different widths, and a fiendish specially constructed
> file, it could be that when you scroll the buffer a single screen line to
> deal with a break between W1 and W2, you create the same problem between
> W2 and W3.  In such a scenario, you might end up scrolling the buffer
> quite a long way in the search for no "broken" continued lines at either
> boundary.  With such a file there might be NO position where there isn't
> a break.

Are you talking about lines so long that they take more than one
window-full to display?  If so, let's not bother about those for the
moment.  Do you see such problem with reasonably short lines?

> In real life?  I imagine it's quite common to use Follow Mode with very
> narrow windows (say, four at ~60 wide) when the annoyance of continued
> lines is less important than seeing all of a long function on the screen
> at once.  There will then be lots of continued lines, and 3 window
> boundaries, W1-W2, W2-W3, W3-W4.

I don't expect "lots", except in text buffers.  Program sources
usually have shorter lines.  In any case, it would be rare to see
lines that take more than 2 screen lines.

> If we were to go this route (of repositioning to avoid line breaks
> between follow windows), there would have to be a limit on how far one
> could scroll, with a value such as 3.

In what units?  Screen lines?  Why only 3?

Re: "... the window start at a meaningless point within a line."

[Alan Mackenzie]

Hello, Eli.

On Thu, Oct 01, 2015 at 01:16:03PM +0300, Eli Zaretskii wrote:
> > Date: Thu, 1 Oct 2015 09:41:38 +0000
> > Cc: emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > > > In the Elisp manual, page "Window Start and End", it's described how, if
> > > > in (set-window-start x) x is a position not at the start of a line, the
> > > > display engine will, instead of x, choose some value near x as the
> > > > window start.  Is this really necessary?

> > > The reason for this is that we want the physical beginning of the line
> > > (i.e. the character after a newline or at bob) always be aligned at
> > > the left edge of the window's text area, even if it is currently out
> > > of view.  This prevents annoying horizontal scrolling of long lines
> > > when the beginning of the line comes into view.

> > OK.  But we're really talking here about where continuation lines start,
> > not physical BOLs.

> The display engine always starts from the physical BOL when it lays
> out buffer text.  That's because only at a physical BOL it knows that
> the window-relative X coordinate is zero.  Otherwise, there's no
> anchor for it to start at.

OK, so such a change would be, at least somewhat, non-trivial.  Could the
supplied window start position, as in (set-window-start line-middle-pos
nil t), not count as such an anchor?

> Invalidating this basic assumption will cause strange effects,
> including the horizontal scrolling I mentioned.

This horizontal scrolling (of a long line at the top of W3, when a
scroll-down brings the beginning of the line onto the window) would be
precisely what's wanted here.  This would ensure that every character on
the line would be displayed exactly once, both when the line straddles W2 and
W3, and after it is scrolled entirely onto W3.

> > > But let me turn the table and ask you: how difficult would it be to
> > > make sure follow-mode never lets a window end in the middle of a
> > > continued line?

> > In the worst case, very difficult.  Indeed, with three follow windows,
> > all of slightly different widths, and a fiendish specially constructed
> > file, it could be that when you scroll the buffer a single screen line to
> > deal with a break between W1 and W2, you create the same problem between
> > W2 and W3.  In such a scenario, you might end up scrolling the buffer
> > quite a long way in the search for no "broken" continued lines at either
> > boundary.  With such a file there might be NO position where there isn't
> > a break.

> Are you talking about lines so long that they take more than one
> window-full to display?  If so, let's not bother about those for the
> moment.  Do you see such problem with reasonably short lines?

No, I was just thinking of ordinary 100 character lines in ordinary
windows.

Let me outline the problem I saw: I have a command `scrolldown-n', bound
to S-<up>.  With point in W3, S-<up> was failing to scroll the screen.
The cause was the split line making (window-end W2) different from
(window-start W3), which messed up Follow Mode's window alignment
routines.

Another (lesser) problem is moving point from W2 to W3 with C-x o
sometimes causes W1 and W2 to scroll up one line; this has the same
cause.  It is suboptimal.

> > In real life?  I imagine it's quite common to use Follow Mode with very
> > narrow windows (say, four at ~60 wide) when the annoyance of continued
> > lines is less important than seeing all of a long function on the screen
> > at once.  There will then be lots of continued lines, and 3 window
> > boundaries, W1-W2, W2-W3, W3-W4.

> I don't expect "lots", except in text buffers.  Program sources
> usually have shorter lines.  In any case, it would be rare to see
> lines that take more than 2 screen lines.

> > If we were to go this route (of repositioning to avoid line breaks
> > between follow windows), there would have to be a limit on how far one
> > could scroll, with a value such as 3.

> In what units?  Screen lines?  Why only 3?

Yes, 3 screen lines.  With a command like `scrolldown-n', or C-u 1
<PageUp>, the user is requesting a single line scroll.  Scrolling more
than this, even 2 or 3 lines, would be puzzling and irritating.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: "... the window start at a meaningless point within a line."

[Eli Zaretskii]

> Date: Thu, 1 Oct 2015 11:02:04 +0000
> Cc: emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > The display engine always starts from the physical BOL when it lays
> > out buffer text.  That's because only at a physical BOL it knows that
> > the window-relative X coordinate is zero.  Otherwise, there's no
> > anchor for it to start at.
> 
> OK, so such a change would be, at least somewhat, non-trivial.  Could the
> supplied window start position, as in (set-window-start line-middle-pos
> nil t), not count as such an anchor?

It cannot act as an anchor, because the horizontal X coordinate is not
(and cannot be) known by the caller.

> > Invalidating this basic assumption will cause strange effects,
> > including the horizontal scrolling I mentioned.
> 
> This horizontal scrolling (of a long line at the top of W3, when a
> scroll-down brings the beginning of the line onto the window) would be
> precisely what's wanted here.

Then go ahead and make the change.  The display engine will cope.  I'm
not sure what users will say, but that's not my problem here ;-)

> > > In the worst case, very difficult.  Indeed, with three follow windows,
> > > all of slightly different widths, and a fiendish specially constructed
> > > file, it could be that when you scroll the buffer a single screen line to
> > > deal with a break between W1 and W2, you create the same problem between
> > > W2 and W3.  In such a scenario, you might end up scrolling the buffer
> > > quite a long way in the search for no "broken" continued lines at either
> > > boundary.  With such a file there might be NO position where there isn't
> > > a break.
> 
> > Are you talking about lines so long that they take more than one
> > window-full to display?  If so, let's not bother about those for the
> > moment.  Do you see such problem with reasonably short lines?
> 
> No, I was just thinking of ordinary 100 character lines in ordinary
> windows.

Then I don't see the problem.  yes, you will need to scroll several
lines, but why is that a problem?

> Let me outline the problem I saw: I have a command `scrolldown-n', bound
> to S-<up>.  With point in W3, S-<up> was failing to scroll the screen.
> The cause was the split line making (window-end W2) different from
> (window-start W3), which messed up Follow Mode's window alignment
> routines.

I understood.  If you never break in the middle of a continued line,
this should not happen.

> Another (lesser) problem is moving point from W2 to W3 with C-x o
> sometimes causes W1 and W2 to scroll up one line; this has the same
> cause.  It is suboptimal.

IIUC, you will not be able to fix this.  Changes near the boundary
between 2 windows will always be likely to cause some scrolling like
that.

> > > If we were to go this route (of repositioning to avoid line breaks
> > > between follow windows), there would have to be a limit on how far one
> > > could scroll, with a value such as 3.
> 
> > In what units?  Screen lines?  Why only 3?
> 
> Yes, 3 screen lines.  With a command like `scrolldown-n', or C-u 1
> <PageUp>, the user is requesting a single line scroll.  Scrolling more
> than this, even 2 or 3 lines, would be puzzling and irritating.

Not sure if horizontal scrolling will be more or less irritating, but
I guess we will find out soon.

Re: "... the window start at a meaningless point within a line."

[Alan Mackenzie]

Hello, Eli

On Thu, Oct 01, 2015 at 03:03:09PM +0300, Eli Zaretskii wrote:
> > Date: Thu, 1 Oct 2015 11:02:04 +0000
> > Cc: emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

[ .... ]

> > This horizontal scrolling (of a long line at the top of W3, when a
> > scroll-down brings the beginning of the line onto the window) would be
> > precisely what's wanted here.

> Then go ahead and make the change.  The display engine will cope.  I'm
> not sure what users will say, but that's not my problem here ;-)

Underway.  Thus far, I've managed to get all the characters in the broken
line displayed "correctly".  That was by adding the extra optional
parameter to set-window-start, an extra field to struct window, and
testing for it in compute_window_start_on_continuation_line.  I can
scroll the windows, mostly, and this works.  So far, so good.

But vertical_motion doesn't work properly.  (vertical-motion 0) puts
point where Emacs thinks the line start ought to be, not where it
actually is.  I'm looking into this.  There may be quite a few places in
the code like this.

[ .... ]

> > Another (lesser) problem is moving point from W2 to W3 with C-x o
> > sometimes causes W1 and W2 to scroll up one line; this has the same
> > cause.  It is suboptimal.

> IIUC, you will not be able to fix this.  Changes near the boundary
> between 2 windows will always be likely to cause some scrolling like
> that.

:-)

> > > > If we were to go this route (of repositioning to avoid line breaks
> > > > between follow windows), there would have to be a limit on how far one
> > > > could scroll, with a value such as 3.

> > > In what units?  Screen lines?  Why only 3?

> > Yes, 3 screen lines.  With a command like `scrolldown-n', or C-u 1
> > <PageUp>, the user is requesting a single line scroll.  Scrolling more
> > than this, even 2 or 3 lines, would be puzzling and irritating.

> Not sure if horizontal scrolling will be more or less irritating, but
> I guess we will find out soon.

We will, hopefully.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: "... the window start at a meaningless point within a line."

[Alan Mackenzie]

Hello, Eli.

On Thu, Oct 01, 2015 at 03:03:09PM +0300, Eli Zaretskii wrote:
> > Date: Thu, 1 Oct 2015 11:02:04 +0000
> > Cc: emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > > The display engine always starts from the physical BOL when it lays
> > > out buffer text.  That's because only at a physical BOL it knows that
> > > the window-relative X coordinate is zero.  Otherwise, there's no
> > > anchor for it to start at.

> > OK, so such a change would be, at least somewhat, non-trivial.  Could the
> > supplied window start position, as in (set-window-start line-middle-pos
> > nil t), not count as such an anchor?

> It cannot act as an anchor, because the horizontal X coordinate is not
> (and cannot be) known by the caller.

> > > Invalidating this basic assumption will cause strange effects,
> > > including the horizontal scrolling I mentioned.

> > This horizontal scrolling (of a long line at the top of W3, when a
> > scroll-down brings the beginning of the line onto the window) would be
> > precisely what's wanted here.

> Then go ahead and make the change.  The display engine will cope.  I'm
> not sure what users will say, but that's not my problem here ;-)

As expected, it has been more difficult than expected.  Getting
`vertical-motion' working right was a game of "whac-a-mole".  I suspect I
still have to modify one or two other primitives, such as
`compute-motion'.

I have a working proof of concept in a patch below.  It is not ready for
committing for several reasons.  One is that I haven't fully grasped what
SAVE_IT and RESTORE_IT do with the third parameter (bidi data, I think),
so I've just used SAVE_IT everywhere without much thought.  Indeed, I
don't know any languages which aren't L->R.  I was a bit surprised at
having to write my own `line-end-position' which takes a position as
parameter.  Maybe I should add this as an &optional parameter to
`line-end-position' and `line-beginning-position'.

To exercise the patch below, it is best to bind <PageUp> and <PageDown>
to follow-scroll-down/up.  They just work better in follow mode, and are
identical to the standard functions when follow mode is not active.

Load a source file (say, an elisp one) with longish lines into a buffer,
and with C-x 3, make 2, 3, or 4 side by side windows.  Adjust the frame's
and/or windows' boundaries so that there are adjacent windows with
unequal widths (as reported by window-body-width); M-{ and M-} are your
friends, here.  Do M-x follow-mode.  In the course of experimentation, it
is pertinent to have the LH window sometimes narrower than the RH one,
sometimes wider.

Scroll the buffer such that a long line straddles the border between the
two windows of unequal width.  Note that there are no missing characters,
and no repeated characters.  Note also that `vertical-motion', and
commands that use it, work properly (modulo remaining bugs) near the top
of the RH window.

Here's the prototype change:


Fix Follow Mode to handle long lines split between windows properly.

1. Amend the windows code to allow a window to start at an arbitrary
point,
rather than the nearest BOL to it.
2. Enhance, and make visible, "struct it" handling code in xdisp.c.

src/window.h (struct window): Add new field exact_start.
src/window.c (Fset_window_start): Amend code and doc wrt new parameter
`exactstart".
(Fdelete_other_windows_internal, set_window_buffer,
window_scroll_pixel_based)
(window_scroll_line_based, Frecenter, Fmove_to_window_line): Set
exact_start
to false after certain window manipulations.
(recenter): new variable non_exact_start.  Use it to check whether a
non-null
recentering has occurred, and only when so, set window.start etc.
(Fwindow_test_temp): A temporary diagnostic function which displays
window.start and window.exact_start.
(syms_of_window): Include Swindow_test_dump.

src/dispextern.h (SAVE_IT, RESTORE_IT): Move here from xdisp.c.
(forward_to_next_display_line_start,
get_window_start_on_continuation_line)
(reseat_at_window_start): New declarations.
src/xdisp.c (SAVE_IT, RESTORE_IT): Removed (moved to dispextern.h).
(forward_to_next_display_line_start): New function.
(reseat_at_window_start): New function.
(get_window_start_on_continuation_line): New function, extracted from ...
(compute_window_start_on_continuation_line): Now calls the above
function.

src/indent.c (pos_line_end_position): New function.
(maybe_move_to_exact_bol): New function.
(Fvertical_motion): Save state of w->exact_start when "lending" the
window to
another buffer.  When the semi-final position of `it' is on the first
text
line of a window with exact_start set, correct it's position to a visible
BOL
by calling maybe_move_to_exact_bol.

lisp/follow.el (follow-select-if-visible-from-first, follow-redisplay):
Call
set-window-start with `exactstart' parameter set.



diff --git a/lisp/follow.el b/lisp/follow.el
index 938c59e..d2caf4b 100644
--- a/lisp/follow.el
+++ b/lisp/follow.el
@@ -993,7 +993,7 @@ follow-select-if-visible-from-first
 	(save-window-excursion
 	  (let ((windows windows))
 	    (while (and (not win) windows)
-	      (set-window-start (car windows) (point) 'noforce)
+	      (set-window-start (car windows) (point) 'noforce t)
 	      (setq end-pos-end-p (follow-calc-win-end (car windows)))
 	      (goto-char (car end-pos-end-p))
 	      ;; Visible, if dest above end, or if eob is visible
@@ -1046,7 +1046,7 @@ follow-redisplay
 			windows try-first-start win old-win-start)))))
     (dolist (w windows)
       (unless (and preserve-win (eq w win))
-	(set-window-start w start))
+	(set-window-start w start nil t))
       (setq start (car (follow-calc-win-end w))))))
 
 (defun follow-estimate-first-window-start (windows win start)
diff --git a/src/dispextern.h b/src/dispextern.h
index e44b70b..5da365c 100644
--- a/src/dispextern.h
+++ b/src/dispextern.h
@@ -2679,6 +2679,29 @@ struct it
       inhibit_free_realized_faces = true;		\
   } while (false)
 
+/* SAVE_IT and RESTORE_IT are called when we save a snapshot of the
+   iterator state and later restore it.  This is needed because the
+   bidi iterator on bidi.c keeps a stacked cache of its states, which
+   is really a singleton.  When we use scratch iterator objects to
+   move around the buffer, we can cause the bidi cache to be pushed or
+   popped, and therefore we need to restore the cache state when we
+   return to the original iterator.  */
+#define SAVE_IT(ITCOPY, ITORIG, CACHE)		\
+  do {						\
+    if (CACHE)					\
+      bidi_unshelve_cache (CACHE, true);	\
+    ITCOPY = ITORIG;				\
+    CACHE = bidi_shelve_cache ();		\
+  } while (false)
+
+#define RESTORE_IT(pITORIG, pITCOPY, CACHE)	\
+  do {						\
+    if (pITORIG != pITCOPY)			\
+      *(pITORIG) = *(pITCOPY);			\
+    bidi_unshelve_cache (CACHE, false);		\
+    CACHE = NULL;				\
+  } while (false)
+
 /* Bit-flags indicating what operation move_it_to should perform.  */
 
 enum move_operation_enum
@@ -3217,6 +3240,7 @@ void init_iterator (struct it *, struct window *, ptrdiff_t,
 void init_iterator_to_row_start (struct it *, struct window *,
                                  struct glyph_row *);
 void start_display (struct it *, struct window *, struct text_pos);
+bool forward_to_next_display_line_start (struct it *it);
 void move_it_vertically (struct it *, int);
 void move_it_vertically_backward (struct it *, int);
 void move_it_by_lines (struct it *, ptrdiff_t);
@@ -3224,6 +3248,7 @@ void move_it_past_eol (struct it *);
 void move_it_in_display_line (struct it *it,
 			      ptrdiff_t to_charpos, int to_x,
 			      enum move_operation_enum op);
+struct text_pos get_window_start_on_continuation_line (struct window *w);
 bool in_display_vector_p (struct it *);
 int frame_mode_line_height (struct frame *);
 extern bool redisplaying_p;
@@ -3233,6 +3258,8 @@ extern Lisp_Object help_echo_object, previous_help_echo_string;
 extern ptrdiff_t help_echo_pos;
 extern int last_tool_bar_item;
 extern void reseat_at_previous_visible_line_start (struct it *);
+extern void reseat_at_window_start (struct it *it);
+
 extern Lisp_Object lookup_glyphless_char_display (int, struct it *);
 extern ptrdiff_t compute_display_string_pos (struct text_pos *,
 					     struct bidi_string_data *,
diff --git a/src/indent.c b/src/indent.c
index 584f217..95542a7 100644
--- a/src/indent.c
+++ b/src/indent.c
@@ -1957,6 +1957,183 @@ window_column_x (struct window *w, Lisp_Object window,
   return x;
 }
 
+
+/* Get the end of the text line on which POS is. */
+static ptrdiff_t
+pos_line_end_position (struct text_pos *pos)
+{
+  ptrdiff_t pt = PT, pt_byte = PT_BYTE;
+  Lisp_Object eol;
+
+  SET_PT_BOTH (pos->charpos, pos->bytepos);
+  eol = Fline_end_position (Qnil);
+  SET_PT_BOTH (pt, pt_byte);
+  return XINT (eol);
+}
+
+
+/* maybe_move_to_exact_bol
+   
+   IT is at an "xdisp" BOL on the top (text) line on the display,
+   having just been moved NLINES to get there.  Move IT to the correct
+   "exact" BOL.  L0_PHYS_EOL is the end of this (text) line.
+   XDISP_WSTART is the "xdisp" window start.
+
+   An "exact BOL" is the beginning of a line as displayed.  The name
+   "exact" comes from the name of the parameter to `set-window-start'.
+   An "xdisp BOL" is the beginning of a line as calculated by xdisp.c,
+   based on the current window width and the position of the previous
+   beginning of a text line (which isn't invisible).  Exact and xdisp
+   BOLs are in slightly different places when a long text line
+   straddles the boundary between two follow mode windows which are of
+   unequal width.
+
+   The "xdisp" window start is where window-start would be if
+   calculated by xdisp from an earlier \n.  The "exact" window start
+   is the value of w->start (having been set by a call of
+   `set-window-start' with non-nil fourth parameter).  The "xdisp" and
+   "exact" BOLs are those calculated from these window start
+   positions. */
+
+static void
+maybe_move_to_exact_bol (struct it *it, ptrdiff_t nlines,
+			 struct text_pos *xdisp_ws)
+{
+  ptrdiff_t L0_phys_EOL = pos_line_end_position (xdisp_ws);
+  ptrdiff_t beginning, end;
+  ptrdiff_t ws = marker_position (it->w->start);
+  
+  if (nlines <= 0)
+    {
+      beginning = IT_CHARPOS (*it);
+      end = PT;
+    }
+  else
+    {
+      beginning = PT;
+      end = IT_CHARPOS (*it);
+    }
+
+  if (beginning < L0_phys_EOL
+      && end >= ws)
+    {
+      struct it exact_it, xdisp_it; /* ITs pointing to exact and xdisp BOLs. */
+      struct it pre_exact_it, post_exact_it; /* Point to the exact BOLs
+						before/after xdisp_it. */
+      void *cache = NULL;
+      int excess_xdisp_BOLs = 0;
+      bool below = true; /* i.e., exact_it hasn't yet reached `it'. */
+      bool put_it_back_a_BOL = false; /* Supplements the `excess_xdisp_BOLs
+					 mechanism. */
+
+      /* Initialise `exact_it' and `xdisp_it' to the pertinent window starts. */
+      SAVE_IT (exact_it, *it, cache);
+      reseat_at_window_start (&exact_it);
+      exact_it.current_x = exact_it.hpos = 0;
+      SAVE_IT (xdisp_it, exact_it, cache);
+      reseat_at_previous_visible_line_start (&xdisp_it);
+      while (IT_CHARPOS (xdisp_it) < CHARPOS (*xdisp_ws)
+	     && forward_to_next_display_line_start (&xdisp_it));
+      xdisp_it.current_x = xdisp_it.hpos = 0;
+
+      if (IT_CHARPOS (*it) < ws)
+	{
+	  below = false;	/* exact_it is already past `it'. */
+	  SAVE_IT (post_exact_it, exact_it, cache);
+	}
+
+      
+      if (beginning < marker_position (it->w->start))
+	excess_xdisp_BOLs--;
+      if (beginning < CHARPOS (*xdisp_ws)
+	  && CHARPOS (*xdisp_ws) <= end)
+	excess_xdisp_BOLs++;
+
+      /* We "hop" `exact_it' and `xdisp_it' over eachother until we reach the
+	 point where they coincide (usually the next text BOL) or we reach
+	 `end'.	 We calculate the disparity in exact and xdisp BOLs as we go.
+	 Note that exact and xdisp BOLs alternate strictly until the point of
+	 coincidence. */
+      if (CHARPOS (*xdisp_ws) < ws)
+	goto xdisp_ws_is_earlier;
+      while (1)
+	{
+	  if (IT_CHARPOS (exact_it) < IT_CHARPOS (*it))
+	    SAVE_IT (pre_exact_it, exact_it, cache);
+	  if (!forward_to_next_display_line_start (&exact_it))
+	    break;	/* protection against infinite looping. */
+	  if (below
+	      && IT_CHARPOS (exact_it) >= IT_CHARPOS (*it))
+	    {
+	      below = false;
+	      SAVE_IT (post_exact_it, exact_it, cache);
+	    }
+	  if (IT_CHARPOS (exact_it) > end)
+	    break;
+	  if (IT_CHARPOS (exact_it) == IT_CHARPOS (xdisp_it))
+	    {
+	      if (nlines <= 0)
+		excess_xdisp_BOLs--;
+	      put_it_back_a_BOL = true;
+	      break;
+	    }
+	  if (IT_CHARPOS (exact_it) > beginning)
+	    excess_xdisp_BOLs--;
+
+	xdisp_ws_is_earlier:
+	  if (!forward_to_next_display_line_start (&xdisp_it))
+	    break;		/* No infinite looping. */
+	  if (IT_CHARPOS (xdisp_it) > end)
+	    break;
+	  if (IT_CHARPOS (xdisp_it) == IT_CHARPOS (exact_it))
+	    {
+	      excess_xdisp_BOLs++;
+	      if (IT_CHARPOS (xdisp_it) == IT_CHARPOS (*it))
+		{
+		  SAVE_IT (pre_exact_it, exact_it, cache);
+		  SAVE_IT (post_exact_it, exact_it, cache);
+		  forward_to_next_display_line_start (&post_exact_it);
+		}
+	      break;	       /* Will happen after a physical EOL. */
+	    }
+	  if (IT_CHARPOS (xdisp_it) > beginning)
+	    excess_xdisp_BOLs++;
+	}
+
+      /* Decide which exact BOL to move `it' to, depending on `excess_xdisp_BOLs'
+	 and possibly `put_it_back_a_BOL'. */
+      /* First, is `it' off the top of the display? */
+      if (excess_xdisp_BOLs
+	  && IT_CHARPOS (*it) < ws)
+	{
+	  forward_to_next_display_line_start (it);
+	  if (IT_CHARPOS (*it) >= ws)
+	    SAVE_IT (*it, post_exact_it, cache);
+	}
+      /* Is `it' within the initial section of text line 0 on the
+	 screen, in which the "exact" and "xdisp" BOLs differ? */
+      else if (IT_CHARPOS (*it) >= ws
+	       && (IT_CHARPOS (*it) <=
+		   (IT_CHARPOS (exact_it) == IT_CHARPOS (xdisp_it)
+		    ? IT_CHARPOS (exact_it)
+		    : end)))
+	if (excess_xdisp_BOLs == 1 /* Can only happen with nlines > 0. */
+	    || excess_xdisp_BOLs == -1) /* ... with nlines <= 0. */
+	  SAVE_IT (*it, post_exact_it, cache);
+	else
+	  SAVE_IT (*it, pre_exact_it, cache);
+      /* Or is `it' at or beyond the point of coincidence of exact and xdisp
+	 BOLs? */
+      else
+	if (excess_xdisp_BOLs == 0
+	    && put_it_back_a_BOL)
+	  move_it_by_lines (it, -1);
+	else if (excess_xdisp_BOLs != 0
+		 && !put_it_back_a_BOL)
+	  forward_to_next_display_line_start (it);
+    }
+}
+
 DEFUN ("vertical-motion", Fvertical_motion, Svertical_motion, 1, 3, 0,
        doc: /* Move point to start of the screen line LINES lines down.
 If LINES is negative, this means moving up.
@@ -1998,6 +2175,7 @@ whether or not it is currently displayed in some window.  */)
   struct window *w;
   Lisp_Object old_buffer;
   EMACS_INT old_charpos IF_LINT (= 0), old_bytepos IF_LINT (= 0);
+  bool old_exact_start = false;
   Lisp_Object lcols;
   void *itdata = NULL;
 
@@ -2019,9 +2197,11 @@ whether or not it is currently displayed in some window.  */)
       old_buffer = w->contents;
       old_charpos = marker_position (w->pointm);
       old_bytepos = marker_byte_position (w->pointm);
+      old_exact_start = w->exact_start;
       wset_buffer (w, Fcurrent_buffer ());
       set_marker_both (w->pointm, w->contents,
 		       BUF_PT (current_buffer), BUF_PT_BYTE (current_buffer));
+      w->exact_start = false;
     }
 
   if (noninteractive)
@@ -2041,8 +2221,10 @@ whether or not it is currently displayed in some window.  */)
       double start_col;
       int start_x IF_LINT (= 0);
       int to_x = -1;
+      struct text_pos xdisp_ws;
 
       bool start_x_given = !NILP (cur_col);
+
       if (start_x_given)
 	{
 	  start_col = extract_float (cur_col);
@@ -2205,6 +2387,16 @@ whether or not it is currently displayed in some window.  */)
 	    }
 	}
 
+   /* If our actual window start isn't where xdisp.c expects it to
+      be, and IT has ended up on the first text line on the
+      screen, amend IT's position to compensate. */
+      if (w->exact_start)
+	{
+	  xdisp_ws = get_window_start_on_continuation_line (w);
+	  if (marker_position (w->start) != xdisp_ws.charpos)
+	    maybe_move_to_exact_bol (&it, nlines, &xdisp_ws);
+	}
+
       /* Move to the goal column, if one was specified.  If the window
 	 was originally hscrolled, the goal column is interpreted as
 	 an addition to the hscroll amount.  */
@@ -2240,6 +2432,7 @@ whether or not it is currently displayed in some window.  */)
       wset_buffer (w, old_buffer);
       set_marker_both (w->pointm, w->contents,
 		       old_charpos, old_bytepos);
+      w->exact_start = old_exact_start;
     }
 
   return make_number (it.vpos);
diff --git a/src/window.c b/src/window.c
index d61f586..c77de33 100644
--- a/src/window.c
+++ b/src/window.c
@@ -1655,12 +1655,14 @@ Return POS.  */)
   return pos;
 }
 
-DEFUN ("set-window-start", Fset_window_start, Sset_window_start, 2, 3, 0,
+DEFUN ("set-window-start", Fset_window_start, Sset_window_start, 2, 4, 0,
        doc: /* Make display in WINDOW start at position POS in WINDOW's buffer.
 WINDOW must be a live window and defaults to the selected one.  Return
 POS.  Optional third arg NOFORCE non-nil inhibits next redisplay from
-overriding motion of point in order to display at this exact start.  */)
-  (Lisp_Object window, Lisp_Object pos, Lisp_Object noforce)
+overriding motion of point in order to display at this exact
+start. Optional fourth argument EXACTSTART non-nil prevents Emacs from
+repositioning the window to the beginning of a line. */)
+  (Lisp_Object window, Lisp_Object pos, Lisp_Object noforce, Lisp_Object exactstart)
 {
   register struct window *w = decode_live_window (window);
 
@@ -1672,6 +1674,7 @@ overriding motion of point in order to display at this exact start.  */)
   w->update_mode_line = true;
   /* Bug#15957.  */
   w->window_end_valid = false;
+  w->exact_start = !NILP (exactstart);
   wset_redisplay (w);
 
   return pos;
@@ -3079,7 +3082,8 @@ window-start value is reasonable when this function is called.  */)
 	  set_marker_both (w->start, w->contents, pos.bufpos, pos.bytepos);
 	  w->window_end_valid = false;
 	  w->start_at_line_beg = (pos.bytepos == BEGV_BYTE
-				    || FETCH_BYTE (pos.bytepos - 1) == '\n');
+				  || FETCH_BYTE (pos.bytepos - 1) == '\n');
+	  w->exact_start = false;
 	  /* We need to do this, so that the window-scroll-functions
 	     get called.  */
 	  w->optional_new_start = true;
@@ -3268,6 +3272,7 @@ set_window_buffer (Lisp_Object window, Lisp_Object buffer,
       set_marker_restricted (w->start,
 			     make_number (b->last_window_start),
 			     buffer);
+      w->exact_start = false;
       w->start_at_line_beg = false;
       w->force_start = false;
     }
@@ -4819,6 +4824,7 @@ window_scroll_pixel_based (Lisp_Object window, int n, bool whole, bool noerror)
 		    spos = min (XINT (Fline_end_position (Qnil)) + 1, ZV);
 		  set_marker_restricted (w->start, make_number (spos),
 					 w->contents);
+		  w->exact_start = false;
 		  w->start_at_line_beg = true;
 		  w->update_mode_line = true;
 		  /* Set force_start so that redisplay_window will run the
@@ -4872,7 +4878,7 @@ window_scroll_pixel_based (Lisp_Object window, int n, bool whole, bool noerror)
 		dy) * n;
 
       /* Note that move_it_vertically always moves the iterator to the
-         start of a line.  So, if the last line doesn't have a newline,
+	 start of a line.  So, if the last line doesn't have a newline,
 	 we would end up at the start of the line ending at ZV.  */
       if (dy <= 0)
 	{
@@ -4950,7 +4956,7 @@ window_scroll_pixel_based (Lisp_Object window, int n, bool whole, bool noerror)
       vscrolled = true;
     }
 
-  if (! vscrolled)
+  if (!vscrolled && n)
     {
       ptrdiff_t pos = IT_CHARPOS (it);
       ptrdiff_t bytepos;
@@ -4966,6 +4972,7 @@ window_scroll_pixel_based (Lisp_Object window, int n, bool whole, bool noerror)
       /* Set the window start, and set up the window for redisplay.  */
       set_marker_restricted_both (w->start, w->contents, IT_CHARPOS (it),
 				  IT_BYTEPOS (it));
+      w->exact_start = false;
       bytepos = marker_byte_position (w->start);
       w->start_at_line_beg = (pos == BEGV || FETCH_BYTE (bytepos - 1) == '\n');
       w->update_mode_line = true;
@@ -4994,7 +5001,7 @@ window_scroll_pixel_based (Lisp_Object window, int n, bool whole, bool noerror)
 	 in the scroll margin at the top.  */
       move_it_to (&it, PT, -1, -1, -1, MOVE_TO_POS);
       if (IT_CHARPOS (it) == PT && it.current_y >= this_scroll_margin
-          && (NILP (Vscroll_preserve_screen_position)
+	  && (NILP (Vscroll_preserve_screen_position)
 	      || EQ (Vscroll_preserve_screen_position, Qt)))
 	/* We found PT at a legitimate height.  Leave it alone.  */
 	;
@@ -5079,9 +5086,9 @@ window_scroll_pixel_based (Lisp_Object window, int n, bool whole, bool noerror)
 	}
 
       if (charpos == PT && !partial_p
-          && (NILP (Vscroll_preserve_screen_position)
+	  && (NILP (Vscroll_preserve_screen_position)
 	      || EQ (Vscroll_preserve_screen_position, Qt)))
-	/* We found PT before we found the display margin, so PT is ok.  */
+	/* We found PT before we found the display margin, so PT is ok.	 */
 	;
       else if (window_scroll_pixel_based_preserve_y >= 0)
 	{
@@ -5121,7 +5128,7 @@ window_scroll_pixel_based (Lisp_Object window, int n, bool whole, bool noerror)
 
 
 /* Implementation of window_scroll that works based on screen lines.
-   See the comment of window_scroll for parameter descriptions.  */
+   See the comment of window_scroll for parameter descriptions.	 */
 
 static void
 window_scroll_line_based (Lisp_Object window, int n, bool whole, bool noerror)
@@ -5131,7 +5138,7 @@ window_scroll_line_based (Lisp_Object window, int n, bool whole, bool noerror)
      fontification, which in turn can modify buffer text (e.g., if the
      fontification functions replace escape sequences with faces, as
      in `grep-mode-font-lock-keywords').  So we use a marker to record
-     the old point position, to prevent crashes in SET_PT_BOTH.  */
+     the old point position, to prevent crashes in SET_PT_BOTH.	 */
   Lisp_Object opoint_marker = Fpoint_marker ();
   register ptrdiff_t pos, pos_byte;
   register int ht = window_internal_height (w);
@@ -5201,6 +5208,7 @@ window_scroll_line_based (Lisp_Object window, int n, bool whole, bool noerror)
 	max (0, min (scroll_margin, w->total_lines / 4));
 
       set_marker_restricted_both (w->start, w->contents, pos, pos_byte);
+      if (n) w->exact_start = false;
       w->start_at_line_beg = !NILP (bolp);
       w->update_mode_line = true;
       /* Set force_start so that redisplay_window will run
@@ -5214,7 +5222,7 @@ window_scroll_line_based (Lisp_Object window, int n, bool whole, bool noerror)
 	  Fvertical_motion (original_pos, window, Qnil);
 	}
       /* If we scrolled forward, put point enough lines down
-	 that it is outside the scroll margin.  */
+	 that it is outside the scroll margin.	*/
       else if (n > 0)
 	{
 	  int top_margin;
@@ -5244,7 +5252,7 @@ window_scroll_line_based (Lisp_Object window, int n, bool whole, bool noerror)
 	  int bottom_margin;
 
 	  /* If we scrolled backward, put point near the end of the window
-	     but not within the scroll margin.  */
+	     but not within the scroll margin.	*/
 	  SET_PT_BOTH (pos, pos_byte);
 	  tem = Fvertical_motion (make_number (ht - this_scroll_margin), window,
 				  Qnil);
@@ -5289,7 +5297,7 @@ window_scroll_line_based (Lisp_Object window, int n, bool whole, bool noerror)
    screen-full which is defined as the height of the window minus
    next_screen_context_lines.  If N is the symbol `-', scroll.
    DIRECTION may be 1 meaning to scroll down, or -1 meaning to scroll
-   up.  This is the guts of Fscroll_up and Fscroll_down.  */
+   up.	This is the guts of Fscroll_up and Fscroll_down.  */
 
 static void
 scroll_command (Lisp_Object n, int direction)
@@ -5325,7 +5333,7 @@ If ARG is omitted or nil, scroll upward by a near full screen.
 A near full screen is `next-screen-context-lines' less than a full screen.
 Negative ARG means scroll downward.
 If ARG is the atom `-', scroll downward by nearly full screen.
-When calling from a program, supply as argument a number, nil, or `-'.  */)
+When calling from a program, supply as argument a number, nil, or `-'.	*/)
   (Lisp_Object arg)
 {
   scroll_command (arg, 1);
@@ -5338,7 +5346,7 @@ If ARG is omitted or nil, scroll down by a near full screen.
 A near full screen is `next-screen-context-lines' less than a full screen.
 Negative ARG means scroll upward.
 If ARG is the atom `-', scroll upward by nearly full screen.
-When calling from a program, supply as argument a number, nil, or `-'.  */)
+When calling from a program, supply as argument a number, nil, or `-'.	*/)
   (Lisp_Object arg)
 {
   scroll_command (arg, -1);
@@ -5371,12 +5379,12 @@ specifies the window.  This takes precedence over
   else
     {
       /* Nothing specified; look for a neighboring window on the same
-	 frame.  */
+	 frame.	 */
       window = Fnext_window (selected_window, Qnil, Qnil);
 
       if (EQ (window, selected_window))
 	/* That didn't get us anywhere; look for a window on another
-           visible frame.  */
+	   visible frame.  */
 	do
 	  window = Fnext_window (window, Qnil, Qt);
 	while (! FRAME_VISIBLE_P (XFRAME (WINDOW_FRAME (XWINDOW (window))))
@@ -5580,6 +5588,12 @@ and redisplay normally--don't erase and redraw the frame.  */)
   ptrdiff_t charpos, bytepos;
   EMACS_INT iarg IF_LINT (= 0);
   int this_scroll_margin;
+  struct text_pos non_exact_start;
+
+  if (w->exact_start)
+    non_exact_start = get_window_start_on_continuation_line (w);
+  else
+    SET_TEXT_POS_FROM_MARKER (non_exact_start, w->start);
 
   if (buf != current_buffer)
     error ("`recenter'ing a window that does not display current-buffer.");
@@ -5761,16 +5775,20 @@ and redisplay normally--don't erase and redraw the frame.  */)
       bytepos = pos.bytepos;
     }
 
-  /* Set the new window start.  */
-  set_marker_both (w->start, w->contents, charpos, bytepos);
-  w->window_end_valid = false;
+  /* Set the new window start if we actually scrolled.  */
+  if (charpos != CHARPOS (non_exact_start))
+    {
+      set_marker_both (w->start, w->contents, charpos, bytepos);
+      w->exact_start = false;
+      w->window_end_valid = false;
 
-  w->optional_new_start = true;
+      w->optional_new_start = true;
 
-  w->start_at_line_beg = (bytepos == BEGV_BYTE
-			  || FETCH_BYTE (bytepos - 1) == '\n');
+      w->start_at_line_beg = (bytepos == BEGV_BYTE
+			      || FETCH_BYTE (bytepos - 1) == '\n');
 
-  wset_redisplay (w);
+      wset_redisplay (w);
+    }
 
   return Qnil;
 }
@@ -5845,6 +5863,7 @@ zero means top of window, negative means relative to bottom of window.  */)
       int height = window_internal_height (w);
       Fvertical_motion (make_number (- (height / 2)), window, Qnil);
       set_marker_both (w->start, w->contents, PT, PT_BYTE);
+      w->exact_start = false;
       w->start_at_line_beg = !NILP (Fbolp ());
       w->force_start = true;
     }
@@ -7111,6 +7130,22 @@ and scrolling positions.  */)
   return Qnil;
 }
 \f
+
+DEFUN ("window-test-dump", Fwindow_test_dump, Swindow_test_dump, 0, 0, "",
+       doc: /* Dump some critical components of the selected window to `message'.*/)
+  ()
+{
+  Lisp_Object window = Fselected_window ();
+  struct window *w = decode_live_window (window);
+  AUTO_STRING (format, "start: %s; exact_start: %s");
+
+  CALLN (Fmessage, format,
+	 w->start,
+	 w->exact_start ? Qt : Qnil);
+  return Qnil;
+}
+
+\f
 void
 init_window_once (void)
 {
@@ -7320,8 +7355,8 @@ pixelwise even if this option is nil.  */);
   window_resize_pixelwise = false;
 
   DEFVAR_BOOL ("fast-but-imprecise-scrolling",
-               Vfast_but_imprecise_scrolling,
-               doc: /* When non-nil, accelerate scrolling operations.
+	       Vfast_but_imprecise_scrolling,
+	       doc: /* When non-nil, accelerate scrolling operations.
 This comes into play when scrolling rapidly over previously
 unfontified buffer regions.  Only those portions of the buffer which
 are actually going to be displayed get fontified.
@@ -7439,6 +7474,7 @@ displayed after a scrolling operation to be somewhat inaccurate.  */);
   defsubr (&Swindow_parameters);
   defsubr (&Swindow_parameter);
   defsubr (&Sset_window_parameter);
+  defsubr (&Swindow_test_dump);
 }
 
 void
diff --git a/src/window.h b/src/window.h
index eaff57e..5084863 100644
--- a/src/window.h
+++ b/src/window.h
@@ -383,6 +383,10 @@ struct window
        window.  */
     bool_bf suspend_auto_hscroll : 1;
 
+    /* True when the position in ->start is the exact window start pos, and
+       is not to be rounded to a beginning of line. */
+    bool_bf exact_start : 1;
+
     /* Amount by which lines of this window are scrolled in
        y-direction (smooth scrolling).  */
     int vscroll;
diff --git a/src/xdisp.c b/src/xdisp.c
index 863d891..b73d227 100644
--- a/src/xdisp.c
+++ b/src/xdisp.c
@@ -511,28 +511,28 @@ bool help_echo_showing_p;
 
 #define TEXT_PROP_DISTANCE_LIMIT 100
 
-/* SAVE_IT and RESTORE_IT are called when we save a snapshot of the
-   iterator state and later restore it.  This is needed because the
-   bidi iterator on bidi.c keeps a stacked cache of its states, which
-   is really a singleton.  When we use scratch iterator objects to
-   move around the buffer, we can cause the bidi cache to be pushed or
-   popped, and therefore we need to restore the cache state when we
-   return to the original iterator.  */
-#define SAVE_IT(ITCOPY, ITORIG, CACHE)		\
-  do {						\
-    if (CACHE)					\
-      bidi_unshelve_cache (CACHE, true);	\
-    ITCOPY = ITORIG;				\
-    CACHE = bidi_shelve_cache ();		\
-  } while (false)
-
-#define RESTORE_IT(pITORIG, pITCOPY, CACHE)	\
-  do {						\
-    if (pITORIG != pITCOPY)			\
-      *(pITORIG) = *(pITCOPY);			\
-    bidi_unshelve_cache (CACHE, false);		\
-    CACHE = NULL;				\
-  } while (false)
+/* /\* SAVE_IT and RESTORE_IT are called when we save a snapshot of the */
+/*    iterator state and later restore it.  This is needed because the */
+/*    bidi iterator on bidi.c keeps a stacked cache of its states, which */
+/*    is really a singleton.  When we use scratch iterator objects to */
+/*    move around the buffer, we can cause the bidi cache to be pushed or */
+/*    popped, and therefore we need to restore the cache state when we */
+/*    return to the original iterator.  *\/ */
+/* #define SAVE_IT(ITCOPY, ITORIG, CACHE)		\ */
+/*   do {						\ */
+/*     if (CACHE)					\ */
+/*       bidi_unshelve_cache (CACHE, true);	\ */
+/*     ITCOPY = ITORIG;				\ */
+/*     CACHE = bidi_shelve_cache ();		\ */
+/*   } while (false) */
+
+/* #define RESTORE_IT(pITORIG, pITCOPY, CACHE)	\ */
+/*   do {						\ */
+/*     if (pITORIG != pITCOPY)			\ */
+/*       *(pITORIG) = *(pITCOPY);			\ */
+/*     bidi_unshelve_cache (CACHE, false);		\ */
+/*     CACHE = NULL;				\ */
+/*   } while (false) */
 
 /* Functions to mark elements as needing redisplay.  */
 enum { REDISPLAY_SOME = 2};	/* Arbitrary choice.  */
@@ -6239,6 +6239,26 @@ forward_to_next_line_start (struct it *it, bool *skipped_p,
   return newline_found_p;
 }
 
+/* Move IT to the start of the next display line.
+   The return value is true if the beginning of the next line was reached.
+*/
+
+bool
+forward_to_next_display_line_start (struct it *it)
+{
+  enum move_it_result eres;
+  bool result = false;
+  eres = move_it_in_display_line_to (it, ZV, -1, MOVE_TO_POS);
+  if (eres != MOVE_POS_MATCH_OR_ZV)
+    {
+      if (eres != MOVE_LINE_CONTINUED)
+	set_iterator_to_next (it, false);
+      result = true;
+      it->current_x = it->hpos = 0;
+    }
+  return result;
+}
+
 
 /* Set IT's current position to the previous visible line start.  Skip
    invisible text that is so either due to text properties or due to
@@ -6342,6 +6362,17 @@ reseat_at_previous_visible_line_start (struct it *it)
 }
 
 
+/* Reseat iterator IT at the beginning of IT's window.  This is particularly
+   useful when the window's `exact_start' flag is set. */
+
+void
+reseat_at_window_start (struct it *it)
+{
+  SET_TEXT_POS_FROM_MARKER (it->current.pos, it->w->start);
+  reseat (it, it->current.pos, true);
+  CHECK_IT (it);
+}
+
 /* Reseat iterator IT on the next visible line start in the current
    buffer.  ON_NEWLINE_P means position IT on the newline
    preceding the line start.  Skip over invisible text that is so
@@ -15271,13 +15302,13 @@ try_scrolling (Lisp_Object window, bool just_this_one_p,
    from the start of the continued line.  It is the start of the
    screen line with the minimum distance from the old start W->start.  */
 
-static bool
-compute_window_start_on_continuation_line (struct window *w)
+struct text_pos
+get_window_start_on_continuation_line (struct window *w)
 {
   struct text_pos pos, start_pos;
-  bool window_start_changed_p = false;
 
   SET_TEXT_POS_FROM_MARKER (start_pos, w->start);
+  pos = start_pos;
 
   /* If window start is on a continuation line...  Window start may be
      < BEGV in case there's invisible text at the start of the
@@ -15302,7 +15333,7 @@ compute_window_start_on_continuation_line (struct window *w)
       reseat_at_previous_visible_line_start (&it);
 
       /* If the line start is "too far" away from the window start,
-         say it takes too much time to compute a new window start.  */
+	 say it takes too much time to compute a new window start.  */
       if (CHARPOS (start_pos) - IT_CHARPOS (it)
 	  /* PXW: Do we need upper bounds here?  */
 	  < WINDOW_TOTAL_LINES (w) * WINDOW_TOTAL_COLS (w))
@@ -15342,17 +15373,30 @@ compute_window_start_on_continuation_line (struct window *w)
 	      else
 		move_it_by_lines (&it, 1);
 	    }
+	}
+    }
 
-	  /* Set the window start there.  */
+  return pos;
+}
+
+static bool
+compute_window_start_on_continuation_line (struct window *w)
+{
+  struct text_pos pos;
+  bool window_start_changed_p = false;
+
+  if (!w->exact_start)
+    {
+      pos = get_window_start_on_continuation_line (w);
+      if (CHARPOS (pos) != marker_position (w->start))
+	{
 	  SET_MARKER_FROM_TEXT_POS (w->start, pos);
 	  window_start_changed_p = true;
 	}
     }
-
   return window_start_changed_p;
 }
 
-
 /* Try cursor movement in case text has not changed in window WINDOW,
    with window start STARTP.  Value is
 

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: "... the window start at a meaningless point within a line."

[Eli Zaretskii]

> Date: Thu, 15 Oct 2015 18:16:43 +0000
> Cc: emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> As expected, it has been more difficult than expected.

It always is.

> I suspect I still have to modify one or two other primitives, such
> as `compute-motion'.

Why do you need to touch compute-motion?  I don't think it's used in
your scenarios.

> I have a working proof of concept in a patch below.  It is not ready for
> committing for several reasons.  One is that I haven't fully grasped what
> SAVE_IT and RESTORE_IT do with the third parameter (bidi data, I think),
> so I've just used SAVE_IT everywhere without much thought.

That argument is where the data is saved and restored.

> --- a/src/dispextern.h
> +++ b/src/dispextern.h
> @@ -2679,6 +2679,29 @@ struct it
>        inhibit_free_realized_faces = true;		\
>    } while (false)
>  
> +/* SAVE_IT and RESTORE_IT are called when we save a snapshot of the
> +   iterator state and later restore it.  This is needed because the
> +   bidi iterator on bidi.c keeps a stacked cache of its states, which
> +   is really a singleton.  When we use scratch iterator objects to
> +   move around the buffer, we can cause the bidi cache to be pushed or
> +   popped, and therefore we need to restore the cache state when we
> +   return to the original iterator.  */
> +#define SAVE_IT(ITCOPY, ITORIG, CACHE)		\
> +  do {						\
> +    if (CACHE)					\
> +      bidi_unshelve_cache (CACHE, true);	\
> +    ITCOPY = ITORIG;				\
> +    CACHE = bidi_shelve_cache ();		\
> +  } while (false)
> +
> +#define RESTORE_IT(pITORIG, pITCOPY, CACHE)	\
> +  do {						\
> +    if (pITORIG != pITCOPY)			\
> +      *(pITORIG) = *(pITCOPY);			\
> +    bidi_unshelve_cache (CACHE, false);		\
> +    CACHE = NULL;				\
> +  } while (false)
> +

I'd prefer these to stay private to xdisp.c.  If you really need them
elsewhere (I don't think so), there are alternative solutions.

> +/* Get the end of the text line on which POS is. */
> +static ptrdiff_t
> +pos_line_end_position (struct text_pos *pos)
> +{
> +  ptrdiff_t pt = PT, pt_byte = PT_BYTE;
> +  Lisp_Object eol;
> +
> +  SET_PT_BOTH (pos->charpos, pos->bytepos);
> +  eol = Fline_end_position (Qnil);
> +  SET_PT_BOTH (pt, pt_byte);
> +  return XINT (eol);
> +}

There are much easier ways to get the end of the line: e.g., scan from
POS for a newline.

> +  if (beginning < L0_phys_EOL
> +      && end >= ws)

Sorry, but this is a non-starter: you cannot compare character
positions with < and >, assuming characters are displayed on a line in
strict logical order.  These comparisons will completely break in
bidirectional text, where character positions change non-linearly with
screen coordinates.

> +      if (IT_CHARPOS (*it) < ws)
> +	{
> +	  below = false;	/* exact_it is already past `it'. */
> +	  SAVE_IT (post_exact_it, exact_it, cache);
> +	}

Same here.

> +      while (1)
> +	{
> +	  if (IT_CHARPOS (exact_it) < IT_CHARPOS (*it))
> +	    SAVE_IT (pre_exact_it, exact_it, cache);
> +	  if (!forward_to_next_display_line_start (&exact_it))
> +	    break;	/* protection against infinite looping. */
> +	  if (below
> +	      && IT_CHARPOS (exact_it) >= IT_CHARPOS (*it))
> +	    {
> +	      below = false;
> +	      SAVE_IT (post_exact_it, exact_it, cache);
> +	    }
> +	  if (IT_CHARPOS (exact_it) > end)
> +	    break;

And here.  And everywhere else.

> +	  if (IT_CHARPOS (exact_it) == IT_CHARPOS (xdisp_it))

Comparisons with == are OK.

I also suspect you don't need all those calls to SAVE_IT all over the
place.  In any case, you are leaking memory here, because there's not
a single RESTORE_IT to free the memory that SAVE_IT allocates.

Can you describe the algorithm of this function in English?  Then
perhaps I could help you rewrite it in bidi-aware way.

Re: "... the window start at a meaningless point within a line."

[Alan Mackenzie]

Hello, Eli.

Thanks for such a rapid reply.

On Thu, Oct 15, 2015 at 11:14:39PM +0300, Eli Zaretskii wrote:
> > Date: Thu, 15 Oct 2015 18:16:43 +0000
> > Cc: emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > I suspect I still have to modify one or two other primitives, such
> > as `compute-motion'.

> Why do you need to touch compute-motion?  I don't think it's used in
> your scenarios.

Because it deals with screen lines and columns.  But looking more closely
at it, it doesn't use struct it's, or anything like that, so perhaps I
won't need to do anything to it.

> > I have a working proof of concept in a patch below.  It is not ready for
> > committing for several reasons.  One is that I haven't fully grasped what
> > SAVE_IT and RESTORE_IT do with the third parameter (bidi data, I think),
> > so I've just used SAVE_IT everywhere without much thought.

> That argument is where the data is saved and restored.

OK.

> I'd prefer these [SAVE_IT and RESTORE_IT] to stay private to xdisp.c.
> If you really need them elsewhere (I don't think so), there are
> alternative solutions.

I think most of the time, I just need to assign one struct it to another.

> > +/* Get the end of the text line on which POS is. */
> > +static ptrdiff_t
> > +pos_line_end_position (struct text_pos *pos)
> > +{
> > +  ptrdiff_t pt = PT, pt_byte = PT_BYTE;
> > +  Lisp_Object eol;
> > +
> > +  SET_PT_BOTH (pos->charpos, pos->bytepos);
> > +  eol = Fline_end_position (Qnil);
> > +  SET_PT_BOTH (pt, pt_byte);
> > +  return XINT (eol);
> > +}

> There are much easier ways to get the end of the line: e.g., scan from
> POS for a newline.

:-)  OK, I'll get rid of that.

> > +  if (beginning < L0_phys_EOL
> > +      && end >= ws)

> Sorry, but this is a non-starter: you cannot compare character
> positions with < and >, assuming characters are displayed on a line in
> strict logical order.  These comparisons will completely break in
> bidirectional text, where character positions change non-linearly with
> screen coordinates.

This just checks that there is some overlap between (beginning end) and
the first text line.

> > +      if (IT_CHARPOS (*it) < ws)
> > +	{
> > +	  below = false;	/* exact_it is already past `it'. */
> > +	  SAVE_IT (post_exact_it, exact_it, cache);
> > +	}

> Same here.

> > +      while (1)
> > +	{
> > +	  if (IT_CHARPOS (exact_it) < IT_CHARPOS (*it))
> > +	    SAVE_IT (pre_exact_it, exact_it, cache);
> > +	  if (!forward_to_next_display_line_start (&exact_it))
> > +	    break;	/* protection against infinite looping. */
> > +	  if (below
> > +	      && IT_CHARPOS (exact_it) >= IT_CHARPOS (*it))
> > +	    {
> > +	      below = false;
> > +	      SAVE_IT (post_exact_it, exact_it, cache);
> > +	    }
> > +	  if (IT_CHARPOS (exact_it) > end)
> > +	    break;

> And here.  And everywhere else.

> > +	  if (IT_CHARPOS (exact_it) == IT_CHARPOS (xdisp_it))

> Comparisons with == are OK.

There must be some bidi compatible order operators which express "earlier
than" etc.  I will be needing these.

> I also suspect you don't need all those calls to SAVE_IT all over the
> place.  In any case, you are leaking memory here, because there's not
> a single RESTORE_IT to free the memory that SAVE_IT allocates.

As I said, straight assignments should do most of the time.  But at the
very end, when I actually change `it', I think some adjustment of the
bidi data will be needed, too.

> Can you describe the algorithm of this function in English?  Then
> perhaps I could help you rewrite it in bidi-aware way.

OK, here goes!  It's a bit long winded, but that's because the function
is little more than an assemblage of special cases.



What maybe_move_to_exact_bol does
---------------------------------

Normally, when set-window-start is called, Emacs adjusts the position of the
window start to the nearest start of display line, calculated from a previous
BOL.  When `et-window-start's EXACT-START argument is non-nil, the window
start is at the exact position supplied.  In this situation, the BO(display)Ls
are in a different place than where xdisp.c expects them to be.  We refer to
these BOLs as "exact BOL"s, and "xdisp BOL"s respectively.

In the following diagrams, these symbols represent various BOLs:
(i) WS is the (exact) window start.
(ii) L2, L3, .... are the subsequent exact BOLs on the first text line.
(iii) B is the xdisp BOL before WS.
(iv) A is the first xdisp BOL after WS.  A2, A3, are subsequent xdisp BOLs.
(v) C is the first BOL where exact and xdisp BOLs coincide (usually after a
  LF).  C2, C3, ... are subsequent BOLs.  They are also called "common" BOLs.

So a typical layout of BOLs at the top of the screen would look something like
this:

  B-----------------WS---A-------------L2---A2-------------\nC--------C2---

Here "xdisp_WS", the place where xdisp would have chosen window start, is at
A.  By contrast, in the following, xdisp_WS would be at B:

  B---WS-----------------A---L2-------------A2---L3---------\nC-------C2---

When Fvertical_motion runs, it first sets its struct it "it" to a BOL, then
possibly moves it to the appropriate column.  Between these two actions,
maybe_move_to_exact_bol reseats `it' from an xdisp BOL to an exact BOL, or
sometimes from a common BOL to an earlier or later BOL.

In maybe_move_to_exact_bol, `beginning' and `end' are PT and `it', in the
canonical order.  Note that `it' will always be at an xdisp (or common) BOL.

maybe_move_to_exact_bol first checks that there is some overlap between the
interval (beginning end) and the first text line on the window.  If there
isn't, the function does nothing.  Otherwise...

The function in essence scans the first text line from WS to C, "counting" the
two sorts of BOL which are contained in (beginning end).  We have two struct
it's, exact_it and xdisp_it, which alternately "hop" forwards over eachother.
When an xdisp BOL is encountered, excess_xdisp_BOLs is incremented; on an
exact BOL, excess_xdisp_BOLs is decremented.  Exact and xdisp BOLs alternate
strictly, so the value of excess_xdisp_BOLs is always -1, 0, or 1.  Note that
a BOL at `beginning' isn't "counted"; one at `end' is.  This is due to the way
Fvertical_motion first moves `it' to the beginning of the current line
before moving it by nlines.

It turns out that when nlines > 0, excess_xdisp_BOLs will end up 0 or 1
(since the final change to the variable, at `it', is an increment).  When
nlines <= 0, the variable will end up -1 or 0 (since the first change, at
the first exact BOL after `it', is a decrement).

At the end of the function, when excess_xdisp_BOLs is -1 or +1, we move `it'
to the exact BOL _after_ `it'.  When it's 0, we move `it' to the exact BOL
_before_ `it' - that's when `it' is on the first text line.  We keep track of
these positions in pre_exact_it and post_exact_it, these variables being moved
forwards at each exact hop.  (They are also incremented when the final xdisp
hop reaches C.)  When `it' is on a subsequent (common) BOL, and
excess_xdisp_BOLs != 0, we advance `it' by one BOL.  There is also a specific
circumstance when we must move `it' to the previous (common) BOL (see example
5 below).

Further notation:
(i) ^ represents PT (point) (which Fvertical_motion hasn't yet changed).
(ii) "it" is the position of Fvertical_motion's struct it `it'.
(iii) T is the "target" - the exact or common BOL we want to end up at.

Here is an example, with nlines > 0, and both PT and `it' are on the first
text line.  The third line in the diagram represents the current value of
excess_xdisp_BOLs.  It's value is only given where it is changed:

    nlines = 2.
1.  B-----------------WS---A-------------L2---A2-------------\nC--------C2---
                      ^                       it               T
                      -    1             0    1

Here, with nlines = 2, Fvertical_motion puts `it' to B, then moves 2 (xdisp)
BOLs forward to A2.  maybe_move_to_exact_bol starts "counting" _after_ PT, and
on reaching A2, excess_xdisp_BOLs has the value 1.  So the function moves `it'
forward to C, our desired target.

In the next example, WS_exact < WS, and nlines < 0:

    nlines = -1
2.  B---WS-----------------A---L2-------------A2---L3---------\nC-------C2---
        T                  it                   ^
    -   -                  -   -1             0

Here, with nlines = -1, Fvertical_motion first moves `it' back to A2, then
moves one line back to A.  maybe_move_to_exact_bol starts "counting" at L2,
and finishes at A2, excess_xdisp_BOLs being 0.  So we move `it' backwards to
WS, our target.

Things get a little more complicated when `end' is at or after C, since the
common BOLs count as both exact and xdisp BOLs.  (Sometimes it is helpful to
think of C as an exact BOL and an xdisp BOL in the same place, but in a
particular order.)  We detect C by comparing exact_it and xdisp_it after every
hop.  We don't count BOLs beyond C because that would be tiring and
unnecessary.

Here is an example with nlines > 0, where we go beyond C:

    nlines = 3
3.  B--------------WS---A------------L2---A2-----------\nC--------C2-----C3
                           ^                                      it,T
                   -    -            -1   0              -1,0

Fvertical_motion moves `it' back to A then forward 3 to C2.  Our target is
also at C2.  When xdisp_it reaches C (and detects it by comparing with
exact_it), it increments excess_xdisp_BOLs to 0, moves pre_exact_it and
post_exact_it forwards to C and C2, and the hopping stops.  excess_xdisp_BOLs
is 0, but `it' is already beyond C.  So we leave `it' where it is, at C2.

In the above example, the exact hop arrives at C before the xdisp hop.  In the
next example, it's the other way around:

    nlines = 3
4.  B---WS---------------A---L2-----------A2---L3--------\nC-------C2------C3
                                ^                                  it,T
    -   -                -   -            1    0           1

Note that when, as here, the exact hop arrives at C second (detecting it), we
only decrement excess_xdisp_BOLs when nlines <= 0 (which it isn't here).  We
do however set put_it_back_a_BOL to true.  put_it_back_a_BOL "cancels"
excess_xdisp_BOLs's value 1, so we don't move `it' from C2.  (Yes, this is an
ugly artifice.)

The next, and final, example shows why we need the ugly boolean
put_it_back_a_BOL:

   nlines = 2
5. B---WS-------------A---L1------\nC----------C2---
                      ^             T          it
   -   -              -   -1        0[,-1]

In this scenario, Fvertical_motion has advanced `it' to C2, but our target is
C.  The normal excess_xdisp_BOLs mechanism, when `it' is already beyond C,
either advances `it' by one BOL or leaves it alone.  Here we need to move `it'
one BOL backwards, to C.  Hence our boolean variable put_it_back_a_BOL, which
will have been set to true when the exact hop arrived at C second.



-- 
Alan Mackenzie (Nuremberg, Germany).

Re: "... the window start at a meaningless point within a line."

[Eli Zaretskii]

> Date: Fri, 16 Oct 2015 09:55:35 +0000
> Cc: emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > Why do you need to touch compute-motion?  I don't think it's used in
> > your scenarios.
> 
> Because it deals with screen lines and columns.  But looking more closely
> at it, it doesn't use struct it's, or anything like that, so perhaps I
> won't need to do anything to it.

AFAIR, it's only used in batch mode, so should not be relevant.

> > I'd prefer these [SAVE_IT and RESTORE_IT] to stay private to xdisp.c.
> > If you really need them elsewhere (I don't think so), there are
> > alternative solutions.
> 
> I think most of the time, I just need to assign one struct it to another.

Let's postpone discussing this until we have a working code that
doesn't break bidi.

> > > +      if (IT_CHARPOS (*it) < ws)
> > > +	{
> > > +	  below = false;	/* exact_it is already past `it'. */
> > > +	  SAVE_IT (post_exact_it, exact_it, cache);
> > > +	}
> 
> > Same here.
> 
> > > +      while (1)
> > > +	{
> > > +	  if (IT_CHARPOS (exact_it) < IT_CHARPOS (*it))
> > > +	    SAVE_IT (pre_exact_it, exact_it, cache);
> > > +	  if (!forward_to_next_display_line_start (&exact_it))
> > > +	    break;	/* protection against infinite looping. */
> > > +	  if (below
> > > +	      && IT_CHARPOS (exact_it) >= IT_CHARPOS (*it))
> > > +	    {
> > > +	      below = false;
> > > +	      SAVE_IT (post_exact_it, exact_it, cache);
> > > +	    }
> > > +	  if (IT_CHARPOS (exact_it) > end)
> > > +	    break;
> 
> > And here.  And everywhere else.
> 
> > > +	  if (IT_CHARPOS (exact_it) == IT_CHARPOS (xdisp_it))
> 
> > Comparisons with == are OK.
> 
> There must be some bidi compatible order operators which express "earlier
> than" etc.  I will be needing these.

Define "earlier than".  Is that on screen, i.e. in the visual order,
or in the buffer, i.e. in the logical order, a.k.a. "the order of
buffer positions"?  These two orders are different for bidirectional
text.

> > I also suspect you don't need all those calls to SAVE_IT all over the
> > place.  In any case, you are leaking memory here, because there's not
> > a single RESTORE_IT to free the memory that SAVE_IT allocates.
> 
> As I said, straight assignments should do most of the time.

As long as you use functions that move the iterator, you cannot safely
assign one 'struct it' to another.  That's because the iterator object
has a (mostly invisible) companion -- the bidi cache, which must be in
sync with the contents of 'struct it'.

> > Can you describe the algorithm of this function in English?  Then
> > perhaps I could help you rewrite it in bidi-aware way.
> 
> OK, here goes!  It's a bit long winded, but that's because the function
> is little more than an assemblage of special cases.

Thanks, I will study it and see what can I propose.

Re: "... the window start at a meaningless point within a line."

[Alan Mackenzie]

Hello, Eli.

On Fri, Oct 16, 2015 at 01:19:09PM +0300, Eli Zaretskii wrote:
> > Date: Fri, 16 Oct 2015 09:55:35 +0000
> > Cc: emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > I think most of the time, I just need to assign one struct it to another.

> Let's postpone discussing this until we have a working code that
> doesn't break bidi.

Good idea!

[ .... ]

> > > Comparisons with == are OK.

> > There must be some bidi compatible order operators which express
> > "earlier than" etc.  I will be needing these.

> Define "earlier than".  Is that on screen, i.e. in the visual order,
> or in the buffer, i.e. in the logical order, a.k.a. "the order of
> buffer positions"?  These two orders are different for bidirectional
> text.

I've no experience with bidi text, or even RTL text.  Could you cite me
an appropriate bidi or RTL file I could play around with, even though I
couldn't read it.  Would I need any special set up with (for example)
fonts, or input methods?  In fact, before asking you to answer these I
should read the fine manual first.

Coming back to the first application of "earlier than" in the function,
it determines whether or not there is overlap between the first text line
on the screen, and the text between PT and `it'.  At the moment, this
check for overlap is done with

  if (beginning < L0_phys_EOL
      && end >= ws)

I can't grasp why this comparison might fail under bidi - all four
variables hold text positions.

What could go wrong with comparisons involving BOL positions?  Am I right
in thinking that in an RTL line, the "beginning of the line", i.e. the
character immediately following a linefeed, would be displayed at the
right hand side of the window, and that subsequent characters would be
displayed to the left of this?

I see that `it' movements from "left to right" are going to move to
successively lower buffer positions, and this will give me trouble.

[ .... ]

> As long as you use functions that move the iterator, you cannot safely
> assign one 'struct it' to another.  That's because the iterator object
> has a (mostly invisible) companion -- the bidi cache, which must be in
> sync with the contents of 'struct it'.

But surely each struct it has its own bidi cache, doesn't it?  That would
be the component bidi_it inside struct it?  Or am I missing something
here?

[ .... ]

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: "... the window start at a meaningless point within a line."

[Eli Zaretskii]

> Date: Fri, 16 Oct 2015 15:19:36 +0000
> Cc: emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > > There must be some bidi compatible order operators which express
> > > "earlier than" etc.  I will be needing these.
> 
> > Define "earlier than".  Is that on screen, i.e. in the visual order,
> > or in the buffer, i.e. in the logical order, a.k.a. "the order of
> > buffer positions"?  These two orders are different for bidirectional
> > text.
> 
> I've no experience with bidi text, or even RTL text.  Could you cite me
> an appropriate bidi or RTL file I could play around with, even though I
> couldn't read it.  Would I need any special set up with (for example)
> fonts, or input methods?  In fact, before asking you to answer these I
> should read the fine manual first.

Just turn on the 'hebrew' input method and start typing.

> Coming back to the first application of "earlier than" in the function,
> it determines whether or not there is overlap between the first text line
> on the screen, and the text between PT and `it'.  At the moment, this
> check for overlap is done with
> 
>   if (beginning < L0_phys_EOL
>       && end >= ws)
> 
> I can't grasp why this comparison might fail under bidi - all four
> variables hold text positions.

With bidirectional text, the character position increases
non-monotonically with screen coordinates.  So the intersection of
buffer position doesn't necessarily mean there's intersection of
screen coordinates, and vice versa.

> What could go wrong with comparisons involving BOL positions?  Am I right
> in thinking that in an RTL line, the "beginning of the line", i.e. the
> character immediately following a linefeed, would be displayed at the
> right hand side of the window, and that subsequent characters would be
> displayed to the left of this?

No.

The issue here is not text being displayed starting from the right
window edge instead of the left edge.  The issue is text that includes
both left-to-right and right-to-left characters.  In this situation, a
character that follows a newline could be anywhere on the next screen
line, even at its end.

> I see that `it' movements from "left to right" are going to move to
> successively lower buffer positions, and this will give me trouble.

No, it's much worse: when it moves, the screen coordinates increase
monotonously, but buffer position does not.  The buffer position can
go up for a while, then jump a head many places and start moving down,
then jump again and go up, etc.  That's what I meant when I said the
buffer position changes non-linearly with screen position.

> > As long as you use functions that move the iterator, you cannot safely
> > assign one 'struct it' to another.  That's because the iterator object
> > has a (mostly invisible) companion -- the bidi cache, which must be in
> > sync with the contents of 'struct it'.
> 
> But surely each struct it has its own bidi cache, doesn't it?

No, there's only one cache.

Re: "... the window start at a meaningless point within a line."

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Fri, 16 Oct 2015 15:19:36 +0000
>> Cc: emacs-devel@gnu.org
>> From: Alan Mackenzie <acm@muc.de>
>> 
>> > > There must be some bidi compatible order operators which express
>> > > "earlier than" etc.  I will be needing these.
>> 
>> > Define "earlier than".  Is that on screen, i.e. in the visual order,
>> > or in the buffer, i.e. in the logical order, a.k.a. "the order of
>> > buffer positions"?  These two orders are different for bidirectional
>> > text.
>> 
>> I've no experience with bidi text, or even RTL text.  Could you cite me
>> an appropriate bidi or RTL file I could play around with, even though I
>> couldn't read it.  Would I need any special set up with (for example)
>> fonts, or input methods?  In fact, before asking you to answer these I
>> should read the fine manual first.
>
> Just turn on the 'hebrew' input method and start typing.

דוד doesn't look all that different from L2R text.

-- 
David Kastrup

Re: "... the window start at a meaningless point within a line."

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Cc: Alan Mackenzie <acm@muc.de>,  emacs-devel@gnu.org
> Date: Fri, 16 Oct 2015 22:46:57 +0200
> 
> > Just turn on the 'hebrew' input method and start typing.

> דוד doesn't look all that different from L2R text.

Yes, that's an example.  But this example is actually not what Alan
should look at, he should look at something like this:

  L2R paragraph with some דוד מלך ישראל R2L text.

The reason is that the iterator geometry is always L2R; screen lines
in an R2L paragraph are produced "behind the back" of the iterator
object.  So all the move_it_* functions think the line begins at the
left window margin, like the above example.

To see the non-linear change in buffer position with the screen
coordinates, use C-f to move through the above line.  C-f moves
forward in the strict logical order of buffer positions, so the
non-linear way the cursor moves on the screen will demonstrate the
problem, i.e. why it is wrong to compare buffer positions to decide
which character is to the left of which on the screen.

Re: "... the window start at a meaningless point within a line."

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>> Cc: Alan Mackenzie <acm@muc.de>,  emacs-devel@gnu.org
>> Date: Fri, 16 Oct 2015 22:46:57 +0200
>> 
>> > Just turn on the 'hebrew' input method and start typing.
>
>> דוד doesn't look all that different from L2R text.
>
> Yes, that's an example.  But this example is actually not what Alan
> should look at, he should look at something like this:
>
>   L2R paragraph with some דוד מלך ישראל R2L text.

Ah, I was just joking about "David" being a palindrome once you throw
out the vowels.  Admittedly, a barely literate joke.  It's my guess one
would not even put back the vowels in Yiddish which is spelled more or
less phonetically with the exception of (non-modern) Hebrew loan words.
But I'm too lazy to look it up right now.

-- 
David Kastrup

Re: "... the window start at a meaningless point within a line."

[Alan Mackenzie]

On Fri, Oct 16, 2015 at 08:26:08PM +0300, Eli Zaretskii wrote:
> > Date: Fri, 16 Oct 2015 15:19:36 +0000
> > Cc: emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

[ .... ]

> > I've no experience with bidi text, or even RTL text.  Could you cite me
> > an appropriate bidi or RTL file I could play around with, even though I
> > couldn't read it.  Would I need any special set up with (for example)
> > fonts, or input methods?  In fact, before asking you to answer these I
> > should read the fine manual first.

> Just turn on the 'hebrew' input method and start typing.

Wow!  That is, er,.. different!

[ .... ]

> > I see that `it' movements from "left to right" are going to move to
> > successively lower buffer positions, and this will give me trouble.

> No, it's much worse: when it moves, the screen coordinates increase
> monotonously, but buffer position does not.  The buffer position can
> go up for a while, then jump a head many places and start moving down,
> then jump again and go up, etc.  That's what I meant when I said the
> buffer position changes non-linearly with screen position.

Yes, I see what you mean, now.  I haven't tried out the Hebrew input
method with follow mode yet.  That should be interesting!

> > > As long as you use functions that move the iterator, you cannot safely
> > > assign one 'struct it' to another.  That's because the iterator object
> > > has a (mostly invisible) companion -- the bidi cache, which must be in
> > > sync with the contents of 'struct it'.

> > But surely each struct it has its own bidi cache, doesn't it?

> No, there's only one cache.

OK.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: "... the window start at a meaningless point within a line."

[Eli Zaretskii]

> Date: Fri, 16 Oct 2015 09:55:35 +0000
> Cc: emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
>   B---WS-----------------A---L2-------------A2---L3---------\nC-------C2---
> 
> When Fvertical_motion runs, it first sets its struct it "it" to a BOL, then
> possibly moves it to the appropriate column.  Between these two actions,
> maybe_move_to_exact_bol reseats `it' from an xdisp BOL to an exact BOL, or
> sometimes from a common BOL to an earlier or later BOL.

But why do you need to do that?  IOW, what problem does
maybe_move_to_exact_bol solve, and how that problem is related to what
Fvertical_motion does?

Perhaps start by explaining what problems you discovered in
Fvertical_motion that required you to make those changes.

Thanks.

Re: "... the window start at a meaningless point within a line."

[Alan Mackenzie]

Hello, Eli.

On Fri, Oct 16, 2015 at 08:35:15PM +0300, Eli Zaretskii wrote:
> > Date: Fri, 16 Oct 2015 09:55:35 +0000
> > Cc: emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> >   B---WS-----------------A---L2-------------A2---L3---------\nC-------C2---

> > When Fvertical_motion runs, it first sets its struct it "it" to a BOL, then
> > possibly moves it to the appropriate column.  Between these two actions,
> > maybe_move_to_exact_bol reseats `it' from an xdisp BOL to an exact BOL, or
> > sometimes from a common BOL to an earlier or later BOL.

> But why do you need to do that?  IOW, what problem does
> maybe_move_to_exact_bol solve, and how that problem is related to what
> Fvertical_motion does?

> Perhaps start by explaining what problems you discovered in
> Fvertical_motion that required you to make those changes.

Without maybe_move_to_exact_bol, vertical-motion, when it lands near
the top of the window, leaves the cursor not at column 0 on the screen.

So, for example, with point at C, unmodified (vertical-motion -1) takes
us to A2, when we want to go to L3.

> Thanks.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: "... the window start at a meaningless point within a line."

[Eli Zaretskii]

> Date: Fri, 16 Oct 2015 18:12:50 +0000
> Cc: emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > But why do you need to do that?  IOW, what problem does
> > maybe_move_to_exact_bol solve, and how that problem is related to what
> > Fvertical_motion does?
> 
> > Perhaps start by explaining what problems you discovered in
> > Fvertical_motion that required you to make those changes.
> 
> Without maybe_move_to_exact_bol, vertical-motion, when it lands near
> the top of the window, leaves the cursor not at column 0 on the screen.
> 
> So, for example, with point at C, unmodified (vertical-motion -1) takes
> us to A2, when we want to go to L3.

So all you need is make a correction after Fvertical_motion did its
thing, is that right?

Will the following algorithm work:

  . compute the horizontal difference, in pixels, between the position
    which xdisp would use as window-start and the actual window-start
    (the value should always be positive); let's call the result N

  . let Fvertical_motion do its thing exactly as it does now

  . move N more pixels to the right, i.e. in the direction of
    increasing the X coordinate, or to the end of line, if it ends
    before that coordinate

If this should work, then all you need is to implement the 1st bullet,
which is very easy, nothing as complicated as your
maybe_move_to_exact_bol.  It should just call move_it_in_display_line
to get to the actual window-start, and save the X coordinate wehen it
gets there.

Re: "... the window start at a meaningless point within a line."

[Eli Zaretskii]

> Date: Fri, 16 Oct 2015 21:23:10 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
>   . compute the horizontal difference, in pixels, between the position
>     which xdisp would use as window-start and the actual window-start
>     (the value should always be positive); let's call the result N
> 
>   . let Fvertical_motion do its thing exactly as it does now
> 
>   . move N more pixels to the right, i.e. in the direction of
>     increasing the X coordinate, or to the end of line, if it ends
>     before that coordinate
> 
> If this should work, then all you need is to implement the 1st bullet,
> which is very easy, nothing as complicated as your
> maybe_move_to_exact_bol.  It should just call move_it_in_display_line
> to get to the actual window-start, and save the X coordinate wehen it
> gets there.

Btw, it might be a good idea to store the offset between the two
window-start points as part of the window structure, instead of
storing just the flag.  The value of the offset will have to be
recomputed whenever text is scrolled or the window dimensions change,
but I suspect you recompute it in those situations anyway, so all you
need to add is a primitive that stores the value.

Then the 1st bullet above doesn't need to be implemented at all, you
just need to use the value stored in w->window_start_offset in the 3rd
bullet.

Does this make sense?

Re: "... the window start at a meaningless point within a line."

[Alan Mackenzie]

Hello, Eli.

On Fri, Oct 16, 2015 at 09:23:10PM +0300, Eli Zaretskii wrote:
> > Date: Fri, 16 Oct 2015 18:12:50 +0000
> > Cc: emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > > But why do you need to do that?  IOW, what problem does
> > > maybe_move_to_exact_bol solve, and how that problem is related to what
> > > Fvertical_motion does?

> > > Perhaps start by explaining what problems you discovered in
> > > Fvertical_motion that required you to make those changes.

> > Without maybe_move_to_exact_bol, vertical-motion, when it lands near
> > the top of the window, leaves the cursor not at column 0 on the screen.

> > So, for example, with point at C, unmodified (vertical-motion -1) takes
> > us to A2, when we want to go to L3.

> So all you need is make a correction after Fvertical_motion did its
> thing, is that right?

After Fvertical_motion has _almost_ done its thing.  (i.e., after it's
moved by rows, but before applying a column offset, should such have
been requested).

> Will the following algorithm work:

No, it won't.  If it did, I'd be somewhat annoyed, having spent so much
time thinking about it, drawing diagrams, and debugging over the last
week or so.  ;-)

>   . compute the horizontal difference, in pixels, between the position
>     which xdisp would use as window-start and the actual window-start
>     (the value should always be positive); let's call the result N

>   . let Fvertical_motion do its thing exactly as it does now

>   . move N more pixels to the right, i.e. in the direction of
>     increasing the X coordinate, or to the end of line, if it ends
>     before that coordinate

> If this should work, then all you need is to implement the 1st bullet,
> which is very easy, nothing as complicated as your
> maybe_move_to_exact_bol.  It should just call move_it_in_display_line
> to get to the actual window-start, and save the X coordinate wehen it
> gets there.

Here's an example why it won't work:

    nlines = 3
1.  B---WS-------------A---L1------------\nC----------C2---------C3-----
                       ^                              T          it

o - Point starts at A, and (since this is already the start of an xdsip
  line) Fvertical_motion moves it forward three lines to C3.
o - What we want to happen is for point to be first set back to the
  (exact) BOL, WS, then moved forward three lines to C2.
o - So, before applying the N pixel correction, the actual and target
  final positions are already a complete line apart, regardless of the
  number of characters (or pixels) on either of these lines.

(Quick recap of the notation: WS is the "exact" window start, L1 is an
"exact" BOL.  B or A (here, B) is the "xdisp" window start, A is an
"xdisp" BOL.  C, C2, C3 are "common" BOLs.  ^ is point, "it" is
Fvertical_motion's first placement of `it', T is our target BOL.)

By making the pertinent text line just a little bit longer, we could end
up with this (note the extra BOL at A2):

    nlines = 3
2.  B---WS-------------A---L1------------A2-\nC---------C2--------C3---
                       ^                                it,T
o - Now Fvertical_motion puts `it' bang on target.
o - To adjust the final position by N pixels would make it wrong.

I see no alternative to counting BOLs of each type up to the point of
coincidence, C.  It is possible this might be done with more finesse
that I've managed so far, but be done it must.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: "... the window start at a meaningless point within a line."

[Eli Zaretskii]

> Date: Fri, 16 Oct 2015 20:12:38 +0000
> Cc: emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > Will the following algorithm work:
> 
> No, it won't.  If it did, I'd be somewhat annoyed, having spent so much
> time thinking about it, drawing diagrams, and debugging over the last
> week or so.  ;-)
> 
> >   . compute the horizontal difference, in pixels, between the position
> >     which xdisp would use as window-start and the actual window-start
> >     (the value should always be positive); let's call the result N
> 
> >   . let Fvertical_motion do its thing exactly as it does now
> 
> >   . move N more pixels to the right, i.e. in the direction of
> >     increasing the X coordinate, or to the end of line, if it ends
> >     before that coordinate
> 
> > If this should work, then all you need is to implement the 1st bullet,
> > which is very easy, nothing as complicated as your
> > maybe_move_to_exact_bol.  It should just call move_it_in_display_line
> > to get to the actual window-start, and save the X coordinate wehen it
> > gets there.
> 
> Here's an example why it won't work:
> 
>     nlines = 3
> 1.  B---WS-------------A---L1------------\nC----------C2---------C3-----
>                        ^                              T          it
> 
> o - Point starts at A, and (since this is already the start of an xdsip
>   line) Fvertical_motion moves it forward three lines to C3.
> o - What we want to happen is for point to be first set back to the
>   (exact) BOL, WS, then moved forward three lines to C2.
> o - So, before applying the N pixel correction, the actual and target
>   final positions are already a complete line apart, regardless of the
>   number of characters (or pixels) on either of these lines.

Starting from a BOL is a special case that can be handled by special
code.  Will the algorithm I suggested work in all other cases?

> By making the pertinent text line just a little bit longer, we could end
> up with this (note the extra BOL at A2):
> 
>     nlines = 3
> 2.  B---WS-------------A---L1------------A2-\nC---------C2--------C3---
>                        ^                                it,T
> o - Now Fvertical_motion puts `it' bang on target.
> o - To adjust the final position by N pixels would make it wrong.

No, it won't: move_it_in_display_line always stops at the end of the
line, even if the goal X coordinate is not yet reached.  It does not
move to the next screen line, as you seem to assume.

> I see no alternative to counting BOLs of each type up to the point of
> coincidence, C.  It is possible this might be done with more finesse
> that I've managed so far, but be done it must.

To find out whether we moved past a newline, you can call
reseat_at_next_visible_line_start.

My analysis of what you described concluded, perhaps wrongly, that the
essence of the problem is the constant offset between the "actual"
beginning of each screen line and the display engine's idea of that
beginning.  This constant offset is limited to the first physical line
of the window; once we get past the first newline, the offset
disappears.  Is this conclusion correct?

Re: "... the window start at a meaningless point within a line."

[Alan Mackenzie]

Hello, Eli.

On Sat, Oct 17, 2015 at 11:33:44AM +0300, Eli Zaretskii wrote:
> > Date: Fri, 16 Oct 2015 20:12:38 +0000
> > Cc: emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > > Will the following algorithm work:

> > No, it won't.  If it did, I'd be somewhat annoyed, having spent so much
> > time thinking about it, drawing diagrams, and debugging over the last
> > week or so.  ;-)

> > >   . compute the horizontal difference, in pixels, between the position
> > >     which xdisp would use as window-start and the actual window-start
> > >     (the value should always be positive); let's call the result N

Just a point I missed last night: N could be either positive or negative,
depending on whether the "exact" window start is just before or just
after the "xdisp" window start.  A situation where N would be negative
is:

3.  B---------------WS---A--------------L1---A2---------\nC---------C2---

, where the "xdisp" window start would be at A, not B.

(BTW, I'm not all that enthusiastic about my terminology "exact" and
"xdisp" here, and if anybody could come up with better suggestions, I'd
welcome it.  But "exact" and "xdisp" are, at the very least, quite
short.)

> > >   . let Fvertical_motion do its thing exactly as it does now

> > >   . move N more pixels to the right, i.e. in the direction of
> > >     increasing the X coordinate, or to the end of line, if it ends
> > >     before that coordinate

> > > If this should work, then all you need is to implement the 1st bullet,
> > > which is very easy, nothing as complicated as your
> > > maybe_move_to_exact_bol.  It should just call move_it_in_display_line
> > > to get to the actual window-start, and save the X coordinate wehen it
> > > gets there.

> > Here's an example why it won't work:

> >     nlines = 3
> > 1.  B---WS-------------A---L1------------\nC----------C2---------C3-----
> >                        ^                              T          it

> > o - Point starts at A, and (since this is already the start of an xdsip
> >   line) Fvertical_motion moves it forward three lines to C3.
> > o - What we want to happen is for point to be first set back to the
> >   (exact) BOL, WS, then moved forward three lines to C2.
> > o - So, before applying the N pixel correction, the actual and target
> >   final positions are already a complete line apart, regardless of the
> >   number of characters (or pixels) on either of these lines.

> Starting from a BOL is a special case that can be handled by special
> code.  Will the algorithm I suggested work in all other cases?

Starting at a(n xdisp) BOL isn't a special case: anywhere between A
(inclusive) and L1 (exclusive) is equivalent.  I think it's probably more
accurate to say that there are cases that your algorithm will handle
correctly.

For example, in diagram 1, move PT a few characters to the right, so that
it reaches (or passes) L1.  Fvertical_motion's handling of the situation
is the same as before (it still moves back to A, then forward to C3), but
the target BOL has become C3.  In the (common) case where the target BOL
is at or after C, adjusting horizontally by N pixels will always be
wrong.  The adjustment we need is to the next, or the previous (common)
BOL.

My feeling is that there is no general case from which a tiny number of
special cases deviate.  Rather, there are just several special cases.

> > By making the pertinent text line just a little bit longer, we could end
> > up with this (note the extra BOL at A2):

> >     nlines = 3
> > 2.  B---WS-------------A---L1------------A2-\nC---------C2--------C3---
> >                        ^                                it,T
> > o - Now Fvertical_motion puts `it' bang on target.
> > o - To adjust the final position by N pixels would make it wrong.

> No, it won't: move_it_in_display_line always stops at the end of the
> line, even if the goal X coordinate is not yet reached.  It does not
> move to the next screen line, as you seem to assume.

> > I see no alternative to counting BOLs of each type up to the point of
> > coincidence, C.  It is possible this might be done with more finesse
> > that I've managed so far, but be done it must.

> To find out whether we moved past a newline, you can call
> reseat_at_next_visible_line_start.

> My analysis of what you described concluded, perhaps wrongly, that the
> essence of the problem is the constant offset between the "actual"
> beginning of each screen line and the display engine's idea of that
> beginning.  This constant offset is limited to the first physical line
> of the window; once we get past the first newline, the offset
> disappears.  Is this conclusion correct?

I don't think so.  That constant offset, N, is indeed relevant in the
first text line in the window.  But after the first newline, whether or
not we have to move one up or down from Fvertical_motion's position is
critically dependent upon the arrangement of exact and xdisp BOLs in that
first text line.

For example, although diagrams 1 and 2 have the same (positive) N, we
need to shift Fvertical_motion's end position by 1 line in diagram 1, but
not in diagram 2.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: "... the window start at a meaningless point within a line."

[Eli Zaretskii]

> Date: Sat, 17 Oct 2015 11:57:38 +0000
> Cc: emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > > >   . compute the horizontal difference, in pixels, between the position
> > > >     which xdisp would use as window-start and the actual window-start
> > > >     (the value should always be positive); let's call the result N
> 
> Just a point I missed last night: N could be either positive or negative,
> depending on whether the "exact" window start is just before or just
> after the "xdisp" window start.  A situation where N would be negative
> is:
> 
> 3.  B---------------WS---A--------------L1---A2---------\nC---------C2---
> 
> , where the "xdisp" window start would be at A, not B.

No, it is always positive, because in my definition the xdisp start is
at B.  It cannot start at A, because that would put point outside of
the window.  IOW, we always compute the offset relative to the BOL of
a screen line that includes the "actual" window start.

> (BTW, I'm not all that enthusiastic about my terminology "exact" and
> "xdisp" here, and if anybody could come up with better suggestions, I'd
> welcome it.  But "exact" and "xdisp" are, at the very least, quite
> short.)

"Exact" is a problematic word; I've been using "actual" instead.

> > Starting from a BOL is a special case that can be handled by special
> > code.  Will the algorithm I suggested work in all other cases?
> 
> Starting at a(n xdisp) BOL isn't a special case: anywhere between A
> (inclusive) and L1 (exclusive) is equivalent.

After the correction of the definition of the "xdisp" BOL above, do
you still disagree that only starting from a BOL is special?
(Actually, I now think that even that situation isn't special, see
below.)

> > My analysis of what you described concluded, perhaps wrongly, that the
> > essence of the problem is the constant offset between the "actual"
> > beginning of each screen line and the display engine's idea of that
> > beginning.  This constant offset is limited to the first physical line
> > of the window; once we get past the first newline, the offset
> > disappears.  Is this conclusion correct?
> 
> I don't think so.  That constant offset, N, is indeed relevant in the
> first text line in the window.  But after the first newline, whether or
> not we have to move one up or down from Fvertical_motion's position is
> critically dependent upon the arrangement of exact and xdisp BOLs in that
> first text line.

Once again, do you still think this after correcting the definition of
the "xdisp" window start?

Btw, I was wrong in that the correction involves movement to the
right.  It actually calls for a movement to the left, which is
slightly more complicated (move_it_* functions cannot do that, so you
need to go to the beginning of a screen line first), but not too
complex.

So now I think that the problem can be solved as follows:

  . if Fvertical_motion's code without changes ends up before the
    first newline that follows the window-start, you don't need to do
    anything, because Fvertical_motion does exactly what you want in
    that case

  . otherwise Fvertical_motion will either end up on the screen line
    before the goal, or will be on the right screen line, but to the
    right of the goal column, and a correction should be applied by
    moving point to the left

Do you agree?

Re: "... the window start at a meaningless point within a line."

[Eli Zaretskii]

> Date: Sat, 17 Oct 2015 15:34:44 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> So now I think that the problem can be solved as follows:
> 
>   . if Fvertical_motion's code without changes ends up before the
>     first newline that follows the window-start, you don't need to do
>     anything, because Fvertical_motion does exactly what you want in
>     that case
> 
>   . otherwise Fvertical_motion will either end up on the screen line
>     before the goal, or will be on the right screen line, but to the
>     right of the goal column, and a correction should be applied by
>     moving point to the left

Is the following a correct definition of when Fvertical_motion
overshoots, i.e. lands one line below (for LINES > 0) the correct one:
when the offset between the "actual" window-start and the preceding
"xdisp" BOL is greater than the X coordinate of the last character of
the 1st physical line in the window.

Re: "... the window start at a meaningless point within a line."

[Eli Zaretskii]

> Date: Sat, 17 Oct 2015 16:31:05 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> Is the following a correct definition of when Fvertical_motion
> overshoots, i.e. lands one line below (for LINES > 0) the correct one:
> when the offset between the "actual" window-start and the preceding
> "xdisp" BOL is greater than the X coordinate of the last character of
> the 1st physical line in the window.

Another idea: did you try augmenting the X coordinate of the iterator
after it gets back to point at the beginning of Fvertical_motion?
IOW, after this line:

	move_it_to (&it,
		    (!disp_string_at_start_p
		     || FETCH_BYTE (IT_BYTEPOS (it)) == '\n')
		    ? PT
		    : PT - 1,
		    -1, -1, -1, MOVE_TO_POS);

modify it.current_x such that it's relative to the "actual"
window-start.  Then let Fvertical_motion do its job as usual.  Did you
try that?

Re: "... the window start at a meaningless point within a line."

[Alan Mackenzie]

Hello, Eli.

On Sat, Oct 17, 2015 at 05:22:21PM +0300, Eli Zaretskii wrote:
> > Date: Sat, 17 Oct 2015 16:31:05 +0300
> > From: Eli Zaretskii <eliz@gnu.org>
> > Cc: emacs-devel@gnu.org
> > 
> > Is the following a correct definition of when Fvertical_motion
> > overshoots, i.e. lands one line below (for LINES > 0) the correct one:
> > when the offset between the "actual" window-start and the preceding
> > "xdisp" BOL is greater than the X coordinate of the last character of
> > the 1st physical line in the window.

> Another idea: did you try augmenting the X coordinate of the iterator
> after it gets back to point at the beginning of Fvertical_motion?
> IOW, after this line:

> 	move_it_to (&it,
> 		    (!disp_string_at_start_p
> 		     || FETCH_BYTE (IT_BYTEPOS (it)) == '\n')
> 		    ? PT
> 		    : PT - 1,
> 		    -1, -1, -1, MOVE_TO_POS);

> modify it.current_x such that it's relative to the "actual"
> window-start.  Then let Fvertical_motion do its job as usual.  Did you
> try that?

No, I haven't.  But I don't think it could work.  After anchoring `it'
at "actual" WS, it would have to move up or down.  Any time the iterator
is moved to a previous line, it re-anchors itself at a previous _xdisp_
BOL then moves forward, losing the relationship with WS.

So this idea might work with nlines > 0, but couldn't work with nlines
<= 0.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: "... the window start at a meaningless point within a line."

[Eli Zaretskii]

> Date: Sun, 18 Oct 2015 15:00:52 +0000
> Cc: emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > Another idea: did you try augmenting the X coordinate of the iterator
> > after it gets back to point at the beginning of Fvertical_motion?
> > IOW, after this line:
> 
> > 	move_it_to (&it,
> > 		    (!disp_string_at_start_p
> > 		     || FETCH_BYTE (IT_BYTEPOS (it)) == '\n')
> > 		    ? PT
> > 		    : PT - 1,
> > 		    -1, -1, -1, MOVE_TO_POS);
> 
> > modify it.current_x such that it's relative to the "actual"
> > window-start.  Then let Fvertical_motion do its job as usual.  Did you
> > try that?
> 
> No, I haven't.  But I don't think it could work.  After anchoring `it'
> at "actual" WS, it would have to move up or down.  Any time the iterator
> is moved to a previous line, it re-anchors itself at a previous _xdisp_
> BOL then moves forward, losing the relationship with WS.
> 
> So this idea might work with nlines > 0, but couldn't work with nlines
> <= 0.

One bird at a time, okay?  Could you please see if this idea works
with nlines > 0?  If it does, it significantly simplifies the
solution, because we could do something very similar for moving
backward, i.e. apply the correction after each move to a previous
line.

Thanks.

Re: "... the window start at a meaningless point within a line."

[Alan Mackenzie]

Hello, Eli.

On Sun, Oct 18, 2015 at 08:44:46PM +0300, Eli Zaretskii wrote:
> > Date: Sun, 18 Oct 2015 15:00:52 +0000
> > Cc: emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > > Another idea: did you try augmenting the X coordinate of the iterator
> > > after it gets back to point at the beginning of Fvertical_motion?
> > > IOW, after this line:

> > > 	move_it_to (&it,
> > > 		    (!disp_string_at_start_p
> > > 		     || FETCH_BYTE (IT_BYTEPOS (it)) == '\n')
> > > 		    ? PT
> > > 		    : PT - 1,
> > > 		    -1, -1, -1, MOVE_TO_POS);

> > > modify it.current_x such that it's relative to the "actual"
> > > window-start.  Then let Fvertical_motion do its job as usual.  Did you
> > > try that?

> > No, I haven't.  But I don't think it could work.  After anchoring `it'
> > at "actual" WS, it would have to move up or down.  Any time the iterator
> > is moved to a previous line, it re-anchors itself at a previous _xdisp_
> > BOL then moves forward, losing the relationship with WS.

> > So this idea might work with nlines > 0, but couldn't work with nlines
> > <= 0.

> One bird at a time, okay?  Could you please see if this idea works
> with nlines > 0?  If it does, it significantly simplifies the
> solution, because we could do something very similar for moving
> backward, i.e. apply the correction after each move to a previous
> line.

OK, I'll try that.  Just give me a bit of time (I'm not as fast at coding
as some people).

> Thanks.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: "... the window start at a meaningless point within a line."

[Alan Mackenzie]

Hello, Eli.

On Mon, Oct 19, 2015 at 10:27:55AM +0000, Alan Mackenzie wrote:
> On Sun, Oct 18, 2015 at 08:44:46PM +0300, Eli Zaretskii wrote:
> > > Date: Sun, 18 Oct 2015 15:00:52 +0000
> > > Cc: emacs-devel@gnu.org
> > > From: Alan Mackenzie <acm@muc.de>

> > > > Another idea: did you try augmenting the X coordinate of the iterator
> > > > after it gets back to point at the beginning of Fvertical_motion?
> > > > IOW, after this line:

> > > > 	move_it_to (&it,
> > > > 		    (!disp_string_at_start_p
> > > > 		     || FETCH_BYTE (IT_BYTEPOS (it)) == '\n')
> > > > 		    ? PT
> > > > 		    : PT - 1,
> > > > 		    -1, -1, -1, MOVE_TO_POS);

> > > > modify it.current_x such that it's relative to the "actual"
> > > > window-start.  Then let Fvertical_motion do its job as usual.  Did you
> > > > try that?

> > > No, I haven't.  But I don't think it could work.  After anchoring `it'
> > > at "actual" WS, it would have to move up or down.  Any time the iterator
> > > is moved to a previous line, it re-anchors itself at a previous _xdisp_
> > > BOL then moves forward, losing the relationship with WS.

> > > So this idea might work with nlines > 0, but couldn't work with nlines
> > > <= 0.

> > One bird at a time, okay?  Could you please see if this idea works
> > with nlines > 0?  If it does, it significantly simplifies the
> > solution, because we could do something very similar for moving
> > backward, i.e. apply the correction after each move to a previous
> > line.

> OK, I'll try that.  Just give me a bit of time (I'm not as fast at coding
> as some people).

I haven't managed to get anywhere with this.  The code in
Fvertical_motion is already quite complicated, due to the variable
it_overshoot_count, and the various things that make it necessary, like
`before-string' or `after-string' overlays with LFs in them.  If we add
in code to correct for "actual" WS into the melee, the code could not
help becoming even more difficult to understand.

What makes me less enthusiastic about the approach is that it only
covers one case, that with (nlines > 0) where point is already within
the first text line in the window (not before it).

I still feel that my approach, of analysing the buffer after
Fvertical_motion has moved the nlines and correcting it's position, is
the right one.  It separates this complexity from the rest of Fv_m.

I've spent some time looking at bidi.c and SAVE_IT, RESTORE_IT.  In
essence, IIUC, the bidi cache that SAVE_IT saves belongs conceptually to
a struct it, but is too big to be included in it.  It would be possible
to modify this cache mechanism so that several struct it's could use it
simultaneously, but this would make things more complicated, and/or
introduce opportunities for memory leaks and so on.  So ....

Would the following strategy for maybe_move_to_exact_bol work?
o - Execute SAVE_IT on *it just once, right at the start.
o - Calculate the target position using several struct it's, as
  currently coded, ignoring bidi aspects.
o - Execute RESTORE_IT on *it.
o - Move *it to our target position.
o - Set it->current_x and it->hpos to zero.

How about putting maybe_move_to_exact_bol into xdisp.c, allowing it to
use SAVE_IT and RESTORE_IT without exporting these two macros?  In fact,
why not put Fvertical_motion into xdisp.c, too?  It would appear to have
quite a lot to do with display, and not a lot to do with indentation.

Another point I'd appreciate clarification on.  Some while back, for
example, your mail of Fri, 16 Oct 2015 20:26:08 +0300, we were talking
about the order of screen positions varying non-linearly with buffer
positions.  Is it true that if a continuation line begins at buffer
position A, then for x < A, buffer position x will be displayed on a
previous line?  (And also for x > A, x will be displayed on this line or
a subsequent one?)

> > Thanks.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: "... the window start at a meaningless point within a line."

[Alan Mackenzie]

Hello, Eli.

On Tue, Oct 27, 2015 at 01:40:25PM +0000, Alan Mackenzie wrote:
> Another point I'd appreciate clarification on.  Some while back, for
> example, your mail of Fri, 16 Oct 2015 20:26:08 +0300, we were talking
> about the order of screen positions varying non-linearly with buffer
> positions.  Is it true that if a continuation line begins at buffer
> position A, then for x < A, buffer position x will be displayed on a
> previous line?  (And also for x > A, x will be displayed on this line or
> a subsequent one?)

I've just found out the answer for myself.  These relationships, however
desirable they might be, do not hold.  So software which needs to
determine which (display) line a character is on needs to use the
move_it_.... routines, somehow.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: "... the window start at a meaningless point within a line."

[Eli Zaretskii]

> Date: Tue, 27 Oct 2015 17:35:33 +0000
> Cc: emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> On Tue, Oct 27, 2015 at 01:40:25PM +0000, Alan Mackenzie wrote:
> > Another point I'd appreciate clarification on.  Some while back, for
> > example, your mail of Fri, 16 Oct 2015 20:26:08 +0300, we were talking
> > about the order of screen positions varying non-linearly with buffer
> > positions.  Is it true that if a continuation line begins at buffer
> > position A, then for x < A, buffer position x will be displayed on a
> > previous line?  (And also for x > A, x will be displayed on this line or
> > a subsequent one?)
> 
> I've just found out the answer for myself.  These relationships, however
> desirable they might be, do not hold.  So software which needs to
> determine which (display) line a character is on needs to use the
> move_it_.... routines, somehow.

Yes, correct.

Re: "... the window start at a meaningless point within a line."

[Eli Zaretskii]

> Date: Tue, 27 Oct 2015 13:40:25 +0000
> Cc: emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> I haven't managed to get anywhere with this.  The code in
> Fvertical_motion is already quite complicated, due to the variable
> it_overshoot_count, and the various things that make it necessary, like
> `before-string' or `after-string' overlays with LFs in them.  If we add
> in code to correct for "actual" WS into the melee, the code could not
> help becoming even more difficult to understand.
> 
> What makes me less enthusiastic about the approach is that it only
> covers one case, that with (nlines > 0) where point is already within
> the first text line in the window (not before it).

In that case, it probably means I will have to do this.  Please send
the smallest patch that allows window-start to be where you want it,
and doesn't involve vertical-motion, together with some simple test
case to show what doesn't work in the unmodified vertical-motion.  I
will then come up with a fix for that function.

> I still feel that my approach, of analysing the buffer after
> Fvertical_motion has moved the nlines and correcting it's position, is
> the right one.  It separates this complexity from the rest of Fv_m.

I don't want to separate it, because vertical-motion already does
almost everything you need, AFAICT.  What you suggest would mean
significant duplication of code that is already quite tricky.  (Please
note that you for now tried to handle only the relatively simple case
of screen lines produced entirely from buffer text, no display or
overlay strings, no images, no display tables, no composed characters.
Most of the hair in vertical-motion's code is because of these use
cases.)

> I've spent some time looking at bidi.c and SAVE_IT, RESTORE_IT.  In
> essence, IIUC, the bidi cache that SAVE_IT saves belongs conceptually to
> a struct it, but is too big to be included in it.

Not only is it big (and theoretically can be as big as the largest
buffer Emacs can support), its size is not fixed: while the display
engine iterates through buffer text, it constantly enlarges and
shrinks the cache as needed (assuming there's bidirectional text; if
not, the cache is always exactly 1 or 2 slots long).  So it cannot be
part of the it structure.

> Would the following strategy for maybe_move_to_exact_bol work?
> o - Execute SAVE_IT on *it just once, right at the start.
> o - Calculate the target position using several struct it's, as
>   currently coded, ignoring bidi aspects.
> o - Execute RESTORE_IT on *it.
> o - Move *it to our target position.
> o - Set it->current_x and it->hpos to zero.

No, you cannot ignore bidi aspects.  The move_it_* functions work in
visual order, not in logical order.  So the bidi aspects are
inherently part of your use of those functions.

> How about putting maybe_move_to_exact_bol into xdisp.c, allowing it to
> use SAVE_IT and RESTORE_IT without exporting these two macros?

That's OK (assuming we need that function -- it looks too large for
such a simple job).  But that is a relatively minor problem.

> In fact, why not put Fvertical_motion into xdisp.c, too?  It would
> appear to have quite a lot to do with display, and not a lot to do
> with indentation.

I don't like moving such important functions between files, it
complicates forensics ("git annotate" etc.).  We have functions
elsewhere that use xdisp.c functionality, so it's not like indent.c is
the only one.

> Another point I'd appreciate clarification on.  Some while back, for
> example, your mail of Fri, 16 Oct 2015 20:26:08 +0300, we were talking
> about the order of screen positions varying non-linearly with buffer
> positions.  Is it true that if a continuation line begins at buffer
> position A, then for x < A, buffer position x will be displayed on a
> previous line?

By "continuation line" do you mean the 2nd, 3rd, etc. screen line of a
long physical line?  If so, the answer is NO, not in general (although
in practice it's almost always like you describe, even when bidi text
is in the buffer).

> (And also for x > A, x will be displayed on this line or a
> subsequent one?)

Also no.

IOW, the visual order is non-linear both in X and in Y directions.

Re: "... the window start at a meaningless point within a line."

[Alan Mackenzie]

Hello, Eli.

On Tue, Oct 27, 2015 at 08:23:07PM +0200, Eli Zaretskii wrote:
> > Date: Tue, 27 Oct 2015 13:40:25 +0000
> > Cc: emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > I haven't managed to get anywhere with this.  The code in
> > Fvertical_motion is already quite complicated, due to the variable
> > it_overshoot_count, and the various things that make it necessary, like
> > `before-string' or `after-string' overlays with LFs in them.  If we add
> > in code to correct for "actual" WS into the melee, the code could not
> > help becoming even more difficult to understand.

> > What makes me less enthusiastic about the approach is that it only
> > covers one case, that with (nlines > 0) where point is already within
> > the first text line in the window (not before it).

> In that case, it probably means I will have to do this.  Please send
> the smallest patch that allows window-start to be where you want it,
> and doesn't involve vertical-motion, together with some simple test
> case to show what doesn't work in the unmodified vertical-motion.  I
> will then come up with a fix for that function.

Thanks, this is appreciated.  I've just created an up to date master
repository to make sure my stuff still works, and it doesn't.  So I'm
working on it.  Maybe I'll get the stuff to you later on today, or
tomorrow.

> > I still feel that my approach, of analysing the buffer after
> > Fvertical_motion has moved the nlines and correcting it's position, is
> > the right one.  It separates this complexity from the rest of Fv_m.

> I don't want to separate it, because vertical-motion already does
> almost everything you need, AFAICT.  What you suggest would mean
> significant duplication of code that is already quite tricky.  (Please
> note that you for now tried to handle only the relatively simple case
> of screen lines produced entirely from buffer text, no display or
> overlay strings, no images, no display tables, no composed characters.
> Most of the hair in vertical-motion's code is because of these use
> cases.)

I was aware of this.  But getting the simplest case working first seemed
like a good idea..

> > I've spent some time looking at bidi.c and SAVE_IT, RESTORE_IT.  In
> > essence, IIUC, the bidi cache that SAVE_IT saves belongs conceptually to
> > a struct it, but is too big to be included in it.

> Not only is it big (and theoretically can be as big as the largest
> buffer Emacs can support), its size is not fixed: while the display
> engine iterates through buffer text, it constantly enlarges and
> shrinks the cache as needed (assuming there's bidirectional text; if
> not, the cache is always exactly 1 or 2 slots long).  So it cannot be
> part of the it structure.

Understood.

[ .... ]

> > In fact, why not put Fvertical_motion into xdisp.c, too?  It would
> > appear to have quite a lot to do with display, and not a lot to do
> > with indentation.

> I don't like moving such important functions between files, it
> complicates forensics ("git annotate" etc.).  We have functions
> elsewhere that use xdisp.c functionality, so it's not like indent.c is
> the only one.

OK.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: "... the window start at a meaningless point within a line."

[Alan Mackenzie]

[-- Attachment #1: Type: text/plain, Size: 21377 bytes --]

Hello, Eli.

On Tue, Oct 27, 2015 at 08:23:07PM +0200, Eli Zaretskii wrote:
> > Date: Tue, 27 Oct 2015 13:40:25 +0000
> > Cc: emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > I haven't managed to get anywhere with this.  The code in
> > Fvertical_motion is already quite complicated, due to the variable
> > it_overshoot_count, and the various things that make it necessary, like
> > `before-string' or `after-string' overlays with LFs in them.  If we add
> > in code to correct for "actual" WS into the melee, the code could not
> > help becoming even more difficult to understand.

> > What makes me less enthusiastic about the approach is that it only
> > covers one case, that with (nlines > 0) where point is already within
> > the first text line in the window (not before it).

> In that case, it probably means I will have to do this.  Please send
> the smallest patch that allows window-start to be where you want it,
> and doesn't involve vertical-motion, together with some simple test
> case to show what doesn't work in the unmodified vertical-motion.  I
> will then come up with a fix for that function.

[ .... ]

The patch below isn't strictly the minimal one you asked for.  The three
new functions I wrote in xdisp.c I have left in, because you might be
able to use them, and they do no harm, and are easily removed if
they're not useful.  In Fvertical_motion, I've left in the bit that
preserves the w->exact_start flag when the window is "borrowed" by
another buffer.  In window.c, I've left in the diagnostic function
Fwindow_test_dump, which is a way of checking whether the window's
exact_start flag is set.  The patch applies cleanly to the master
repository as of yesterday evening.  If you really do want that minimal
patch instead, get back to me and I will create it.

I've included a small elisp file, utilities.el, which contains things I
found useful whilst testing.  In particular, the commands which scroll
the buffer by 1 or 6 lines, the ones which move point to the top and
bottom of the window, and those which scroll point to the top and bottom
of the window are very helpful.  The commands which enable Follow Mode in
several side by side windows are probably more useful on a tty than in a
GUI.  I've also included a command which just calls `vertical-motion',
and bound it to C-c m.

The file fragment.el is an elisp fragment which contains some long
lines.  There is a particularly long line at L162.

After applying the patch, to test things, in X (?or windows)
o - Start emacs -Q.
o - Load utilities.el.
o - Visit fragment.el.
o - Using the mouse, make the frame wide enough for two side by side
  windows of width 79 and 80.
o - C-x 3.
o - Fiddle about with the mouse, M-{, M-}, until the windows are 79, and
  80 wide (`window-body-width' is your friend, here).
o - Scroll the buffer until L162 straddles window 1 and window 2, with
  two display lines in each window.  (S-<up>/<down>, C-S-<up>/<down>
  from utilies.el are handy, here).

First of all, note that in L162, all characters are displayed exactly
once.  This is the purpose of this change.

With point in W2, note that C-<left> doesn't go to the beginning of
the window, and C-<right> doesn't go to the end.  The actual end
positions are a line out in both cases.  With point in the split line,
note that vertical-motion (C-u <n> C-c m) doesn't go to the right
places, even when the target line is below the split line.  Typing <up>
and <down> also produce interesting effects.

Using M-{ or M-}, make W1 80 wide, and W2 79 wide.  Scroll the windows
up and down, back to the same place, to ensure that Follow Mode has
resynchronised its windows[*].  Now repeat the experiments of the
previous paragraph.  If anything, the results now are even worse.

[*] This is something in Follow Mode which still needs amending.

Perhaps I should write a more formal test spec which covers all the
various cases.

Have fun!

Here is the patch:


Add functionality to inhibit redisplay from moving window start to a BOL.

src/window.h (struct window): Add new field exact_start.
src/window.c (Fset_window_start): Amend code and doc wrt new parameter
`exactstart".
(Fdelete_other_windows_internal, set_window_buffer, window_scroll_pixel_based)
(window_scroll_line_based, Frecenter, Fmove_to_window_line): Set exact_start
to false after certain window manipulations.
(recenter): new variable non_exact_start.  Use it to check whether a non-null
recentering has occurred, and only when so, set window.start etc.
(Fwindow_test_dump): A temporary diagnostic function which displays
window.start and window.exact_start.
(syms_of_window): Include Swindow_test_dump.

src/dispextern.h (forward_to_next_display_line_start)
(get_window_start_on_continuation_line, reseat_at_window_start): New
declarations.
src/xdisp.c (forward_to_next_display_line_start): New function.
(reseat_at_window_start): New function.
(get_window_start_on_continuation_line): New function, extracted from ...
(compute_window_start_on_continuation_line): Now calls the above function.

src/indent.c (Fvertical_motion): Save state of w->exact_start when "lending"
the window to another buffer.

lisp/follow.el (follow-select-if-visible-from-first, follow-redisplay): Call
set-window-start with `exactstart' parameter set.



diff --git a/lisp/follow.el b/lisp/follow.el
index 938c59e..d2caf4b 100644
--- a/lisp/follow.el
+++ b/lisp/follow.el
@@ -993,7 +993,7 @@ follow-select-if-visible-from-first
 	(save-window-excursion
 	  (let ((windows windows))
 	    (while (and (not win) windows)
-	      (set-window-start (car windows) (point) 'noforce)
+	      (set-window-start (car windows) (point) 'noforce t)
 	      (setq end-pos-end-p (follow-calc-win-end (car windows)))
 	      (goto-char (car end-pos-end-p))
 	      ;; Visible, if dest above end, or if eob is visible
@@ -1046,7 +1046,7 @@ follow-redisplay
 			windows try-first-start win old-win-start)))))
     (dolist (w windows)
       (unless (and preserve-win (eq w win))
-	(set-window-start w start))
+	(set-window-start w start nil t))
       (setq start (car (follow-calc-win-end w))))))
 
 (defun follow-estimate-first-window-start (windows win start)
diff --git a/src/dispextern.h b/src/dispextern.h
index e44b70b..a4477ed 100644
--- a/src/dispextern.h
+++ b/src/dispextern.h
@@ -3217,6 +3217,7 @@ void init_iterator (struct it *, struct window *, ptrdiff_t,
 void init_iterator_to_row_start (struct it *, struct window *,
                                  struct glyph_row *);
 void start_display (struct it *, struct window *, struct text_pos);
+bool forward_to_next_display_line_start (struct it *it);
 void move_it_vertically (struct it *, int);
 void move_it_vertically_backward (struct it *, int);
 void move_it_by_lines (struct it *, ptrdiff_t);
@@ -3224,6 +3225,7 @@ void move_it_past_eol (struct it *);
 void move_it_in_display_line (struct it *it,
 			      ptrdiff_t to_charpos, int to_x,
 			      enum move_operation_enum op);
+struct text_pos get_window_start_on_continuation_line (struct window *w);
 bool in_display_vector_p (struct it *);
 int frame_mode_line_height (struct frame *);
 extern bool redisplaying_p;
@@ -3233,6 +3235,8 @@ extern Lisp_Object help_echo_object, previous_help_echo_string;
 extern ptrdiff_t help_echo_pos;
 extern int last_tool_bar_item;
 extern void reseat_at_previous_visible_line_start (struct it *);
+extern void reseat_at_window_start (struct it *it);
+
 extern Lisp_Object lookup_glyphless_char_display (int, struct it *);
 extern ptrdiff_t compute_display_string_pos (struct text_pos *,
 					     struct bidi_string_data *,
diff --git a/src/indent.c b/src/indent.c
index 04837f8..32597bb 100644
--- a/src/indent.c
+++ b/src/indent.c
@@ -1996,6 +1996,7 @@ whether or not it is currently displayed in some window.  */)
   struct window *w;
   Lisp_Object old_buffer;
   EMACS_INT old_charpos IF_LINT (= 0), old_bytepos IF_LINT (= 0);
+  bool old_exact_start = false;
   Lisp_Object lcols;
   void *itdata = NULL;
 
@@ -2017,9 +2018,11 @@ whether or not it is currently displayed in some window.  */)
       old_buffer = w->contents;
       old_charpos = marker_position (w->pointm);
       old_bytepos = marker_byte_position (w->pointm);
+      old_exact_start = w->exact_start;
       wset_buffer (w, Fcurrent_buffer ());
       set_marker_both (w->pointm, w->contents,
 		       BUF_PT (current_buffer), BUF_PT_BYTE (current_buffer));
+      w->exact_start = false;
     }
 
   if (noninteractive)
@@ -2039,8 +2042,10 @@ whether or not it is currently displayed in some window.  */)
       double start_col;
       int start_x IF_LINT (= 0);
       int to_x = -1;
+      struct text_pos xdisp_ws;
 
       bool start_x_given = !NILP (cur_col);
+
       if (start_x_given)
 	{
 	  start_col = extract_float (cur_col);
@@ -2242,6 +2247,7 @@ whether or not it is currently displayed in some window.  */)
       wset_buffer (w, old_buffer);
       set_marker_both (w->pointm, w->contents,
 		       old_charpos, old_bytepos);
+      w->exact_start = old_exact_start;
     }
 
   return make_number (it.vpos);
diff --git a/src/window.c b/src/window.c
index 7c95ff9..b922511 100644
--- a/src/window.c
+++ b/src/window.c
@@ -1666,12 +1666,14 @@ Return POS.  */)
   return pos;
 }
 
-DEFUN ("set-window-start", Fset_window_start, Sset_window_start, 2, 3, 0,
+DEFUN ("set-window-start", Fset_window_start, Sset_window_start, 2, 4, 0,
        doc: /* Make display in WINDOW start at position POS in WINDOW's buffer.
 WINDOW must be a live window and defaults to the selected one.  Return
 POS.  Optional third arg NOFORCE non-nil inhibits next redisplay from
-overriding motion of point in order to display at this exact start.  */)
-  (Lisp_Object window, Lisp_Object pos, Lisp_Object noforce)
+overriding motion of point in order to display at this exact
+start. Optional fourth argument EXACTSTART non-nil prevents Emacs from
+repositioning the window to the beginning of a line. */)
+  (Lisp_Object window, Lisp_Object pos, Lisp_Object noforce, Lisp_Object exactstart)
 {
   register struct window *w = decode_live_window (window);
 
@@ -1683,6 +1685,7 @@ overriding motion of point in order to display at this exact start.  */)
   wset_update_mode_line (w);
   /* Bug#15957.  */
   w->window_end_valid = false;
+  w->exact_start = !NILP (exactstart);
   wset_redisplay (w);
 
   return pos;
@@ -3090,7 +3093,8 @@ window-start value is reasonable when this function is called.  */)
 	  set_marker_both (w->start, w->contents, pos.bufpos, pos.bytepos);
 	  w->window_end_valid = false;
 	  w->start_at_line_beg = (pos.bytepos == BEGV_BYTE
-				    || FETCH_BYTE (pos.bytepos - 1) == '\n');
+				  || FETCH_BYTE (pos.bytepos - 1) == '\n');
+	  w->exact_start = false;
 	  /* We need to do this, so that the window-scroll-functions
 	     get called.  */
 	  w->optional_new_start = true;
@@ -3279,6 +3283,7 @@ set_window_buffer (Lisp_Object window, Lisp_Object buffer,
       set_marker_restricted (w->start,
 			     make_number (b->last_window_start),
 			     buffer);
+      w->exact_start = false;
       w->start_at_line_beg = false;
       w->force_start = false;
     }
@@ -4836,7 +4841,8 @@ window_scroll_pixel_based (Lisp_Object window, int n, bool whole, bool noerror)
 		    spos = min (XINT (Fline_end_position (Qnil)) + 1, ZV);
 		  set_marker_restricted (w->start, make_number (spos),
 					 w->contents);
-		  w->start_at_line_beg = true;
+		  w->exact_start = false;
+                  w->start_at_line_beg = true;
 		  wset_update_mode_line (w);
 		  /* Set force_start so that redisplay_window will run the
 		     window-scroll-functions.  */
@@ -4889,7 +4895,7 @@ window_scroll_pixel_based (Lisp_Object window, int n, bool whole, bool noerror)
 		dy) * n;
 
       /* Note that move_it_vertically always moves the iterator to the
-         start of a line.  So, if the last line doesn't have a newline,
+	 start of a line.  So, if the last line doesn't have a newline,
 	 we would end up at the start of the line ending at ZV.  */
       if (dy <= 0)
 	{
@@ -4967,7 +4973,7 @@ window_scroll_pixel_based (Lisp_Object window, int n, bool whole, bool noerror)
       vscrolled = true;
     }
 
-  if (! vscrolled)
+  if (!vscrolled && n)
     {
       ptrdiff_t pos = IT_CHARPOS (it);
       ptrdiff_t bytepos;
@@ -4983,6 +4989,7 @@ window_scroll_pixel_based (Lisp_Object window, int n, bool whole, bool noerror)
       /* Set the window start, and set up the window for redisplay.  */
       set_marker_restricted_both (w->start, w->contents, IT_CHARPOS (it),
 				  IT_BYTEPOS (it));
+      w->exact_start = false;
       bytepos = marker_byte_position (w->start);
       w->start_at_line_beg = (pos == BEGV || FETCH_BYTE (bytepos - 1) == '\n');
       wset_update_mode_line (w);
@@ -5011,7 +5018,7 @@ window_scroll_pixel_based (Lisp_Object window, int n, bool whole, bool noerror)
 	 in the scroll margin at the top.  */
       move_it_to (&it, PT, -1, -1, -1, MOVE_TO_POS);
       if (IT_CHARPOS (it) == PT && it.current_y >= this_scroll_margin
-          && (NILP (Vscroll_preserve_screen_position)
+	  && (NILP (Vscroll_preserve_screen_position)
 	      || EQ (Vscroll_preserve_screen_position, Qt)))
 	/* We found PT at a legitimate height.  Leave it alone.  */
 	;
@@ -5096,7 +5103,7 @@ window_scroll_pixel_based (Lisp_Object window, int n, bool whole, bool noerror)
 	}
 
       if (charpos == PT && !partial_p
-          && (NILP (Vscroll_preserve_screen_position)
+	  && (NILP (Vscroll_preserve_screen_position)
 	      || EQ (Vscroll_preserve_screen_position, Qt)))
 	/* We found PT before we found the display margin, so PT is ok.  */
 	;
@@ -5218,6 +5225,7 @@ window_scroll_line_based (Lisp_Object window, int n, bool whole, bool noerror)
 	max (0, min (scroll_margin, w->total_lines / 4));
 
       set_marker_restricted_both (w->start, w->contents, pos, pos_byte);
+      if (n) w->exact_start = false;
       w->start_at_line_beg = !NILP (bolp);
       wset_update_mode_line (w);
       /* Set force_start so that redisplay_window will run
@@ -5393,7 +5401,7 @@ specifies the window.  This takes precedence over
 
       if (EQ (window, selected_window))
 	/* That didn't get us anywhere; look for a window on another
-           visible frame.  */
+	   visible frame.  */
 	do
 	  window = Fnext_window (window, Qnil, Qt);
 	while (! FRAME_VISIBLE_P (XFRAME (WINDOW_FRAME (XWINDOW (window))))
@@ -5597,6 +5605,12 @@ and redisplay normally--don't erase and redraw the frame.  */)
   ptrdiff_t charpos, bytepos;
   EMACS_INT iarg IF_LINT (= 0);
   int this_scroll_margin;
+  struct text_pos non_exact_start;
+
+  if (w->exact_start)
+    non_exact_start = get_window_start_on_continuation_line (w);
+  else
+    SET_TEXT_POS_FROM_MARKER (non_exact_start, w->start);
 
   if (buf != current_buffer)
     error ("`recenter'ing a window that does not display current-buffer.");
@@ -5778,16 +5792,20 @@ and redisplay normally--don't erase and redraw the frame.  */)
       bytepos = pos.bytepos;
     }
 
-  /* Set the new window start.  */
-  set_marker_both (w->start, w->contents, charpos, bytepos);
-  w->window_end_valid = false;
+  /* Set the new window start if we actually scrolled.  */
+  if (charpos != CHARPOS (non_exact_start))
+    {
+      set_marker_both (w->start, w->contents, charpos, bytepos);
+      w->exact_start = false;
+      w->window_end_valid = false;
 
-  w->optional_new_start = true;
+      w->optional_new_start = true;
 
-  w->start_at_line_beg = (bytepos == BEGV_BYTE
-			  || FETCH_BYTE (bytepos - 1) == '\n');
+      w->start_at_line_beg = (bytepos == BEGV_BYTE
+			      || FETCH_BYTE (bytepos - 1) == '\n');
 
-  wset_redisplay (w);
+      wset_redisplay (w);
+    }
 
   return Qnil;
 }
@@ -5862,6 +5880,7 @@ zero means top of window, negative means relative to bottom of window.  */)
       int height = window_internal_height (w);
       Fvertical_motion (make_number (- (height / 2)), window, Qnil);
       set_marker_both (w->start, w->contents, PT, PT_BYTE);
+      w->exact_start = false;
       w->start_at_line_beg = !NILP (Fbolp ());
       w->force_start = true;
     }
@@ -7127,6 +7146,22 @@ and scrolling positions.  */)
   return Qnil;
 }
 \f
+
+DEFUN ("window-test-dump", Fwindow_test_dump, Swindow_test_dump, 0, 0, "",
+       doc: /* Dump some critical components of the selected window to `message'.*/)
+  ()
+{
+  Lisp_Object window = Fselected_window ();
+  struct window *w = decode_live_window (window);
+  AUTO_STRING (format, "start: %s; exact_start: %s");
+
+  CALLN (Fmessage, format,
+	 w->start,
+	 w->exact_start ? Qt : Qnil);
+  return Qnil;
+}
+
+\f
 void
 init_window_once (void)
 {
@@ -7336,8 +7371,8 @@ pixelwise even if this option is nil.  */);
   window_resize_pixelwise = false;
 
   DEFVAR_BOOL ("fast-but-imprecise-scrolling",
-               Vfast_but_imprecise_scrolling,
-               doc: /* When non-nil, accelerate scrolling operations.
+	       Vfast_but_imprecise_scrolling,
+	       doc: /* When non-nil, accelerate scrolling operations.
 This comes into play when scrolling rapidly over previously
 unfontified buffer regions.  Only those portions of the buffer which
 are actually going to be displayed get fontified.
@@ -7455,6 +7490,7 @@ displayed after a scrolling operation to be somewhat inaccurate.  */);
   defsubr (&Swindow_parameters);
   defsubr (&Swindow_parameter);
   defsubr (&Sset_window_parameter);
+  defsubr (&Swindow_test_dump);
 }
 
 void
diff --git a/src/window.h b/src/window.h
index eaff57e..5084863 100644
--- a/src/window.h
+++ b/src/window.h
@@ -383,6 +383,10 @@ struct window
        window.  */
     bool_bf suspend_auto_hscroll : 1;
 
+    /* True when the position in ->start is the exact window start pos, and
+       is not to be rounded to a beginning of line. */
+    bool_bf exact_start : 1;
+
     /* Amount by which lines of this window are scrolled in
        y-direction (smooth scrolling).  */
     int vscroll;
diff --git a/src/xdisp.c b/src/xdisp.c
index bdf2d09..a8f38c9 100644
--- a/src/xdisp.c
+++ b/src/xdisp.c
@@ -6275,6 +6275,26 @@ forward_to_next_line_start (struct it *it, bool *skipped_p,
   return newline_found_p;
 }
 
+/* Move IT to the start of the next display line.
+   The return value is true if the beginning of the next line was reached.
+*/
+
+bool
+forward_to_next_display_line_start (struct it *it)
+{
+  enum move_it_result eres;
+  bool result = false;
+  eres = move_it_in_display_line_to (it, ZV, -1, MOVE_TO_POS);
+  if (eres != MOVE_POS_MATCH_OR_ZV)
+    {
+      if (eres != MOVE_LINE_CONTINUED)
+	set_iterator_to_next (it, false);
+      result = true;
+      it->current_x = it->hpos = 0;
+    }
+  return result;
+}
+
 
 /* Set IT's current position to the previous visible line start.  Skip
    invisible text that is so either due to text properties or due to
@@ -6378,6 +6398,17 @@ reseat_at_previous_visible_line_start (struct it *it)
 }
 
 
+/* Reseat iterator IT at the beginning of IT's window.  This is particularly
+   useful when the window's `exact_start' flag is set. */
+
+void
+reseat_at_window_start (struct it *it)
+{
+  SET_TEXT_POS_FROM_MARKER (it->current.pos, it->w->start);
+  reseat (it, it->current.pos, true);
+  CHECK_IT (it);
+}
+
 /* Reseat iterator IT on the next visible line start in the current
    buffer.  ON_NEWLINE_P means position IT on the newline
    preceding the line start.  Skip over invisible text that is so
@@ -15341,13 +15372,13 @@ try_scrolling (Lisp_Object window, bool just_this_one_p,
    from the start of the continued line.  It is the start of the
    screen line with the minimum distance from the old start W->start.  */
 
-static bool
-compute_window_start_on_continuation_line (struct window *w)
+struct text_pos
+get_window_start_on_continuation_line (struct window *w)
 {
   struct text_pos pos, start_pos;
-  bool window_start_changed_p = false;
 
   SET_TEXT_POS_FROM_MARKER (start_pos, w->start);
+  pos = start_pos;
 
   /* If window start is on a continuation line...  Window start may be
      < BEGV in case there's invisible text at the start of the
@@ -15372,7 +15403,7 @@ compute_window_start_on_continuation_line (struct window *w)
       reseat_at_previous_visible_line_start (&it);
 
       /* If the line start is "too far" away from the window start,
-         say it takes too much time to compute a new window start.  */
+	 say it takes too much time to compute a new window start.  */
       if (CHARPOS (start_pos) - IT_CHARPOS (it)
 	  /* PXW: Do we need upper bounds here?  */
 	  < WINDOW_TOTAL_LINES (w) * WINDOW_TOTAL_COLS (w))
@@ -15412,17 +15443,30 @@ compute_window_start_on_continuation_line (struct window *w)
 	      else
 		move_it_by_lines (&it, 1);
 	    }
+	}
+    }
 
-	  /* Set the window start there.  */
+  return pos;
+}
+
+static bool
+compute_window_start_on_continuation_line (struct window *w)
+{
+  struct text_pos pos;
+  bool window_start_changed_p = false;
+
+  if (!w->exact_start)
+    {
+      pos = get_window_start_on_continuation_line (w);
+      if (CHARPOS (pos) != marker_position (w->start))
+	{
 	  SET_MARKER_FROM_TEXT_POS (w->start, pos);
 	  window_start_changed_p = true;
 	}
     }
-
   return window_start_changed_p;
 }
 
-
 /* Try cursor movement in case text has not changed in window WINDOW,
    with window start STARTP.  Value is
 


-- 
Alan Mackenzie (Nuremberg, Germany).


[-- Attachment #2: utilities.el --]
[-- Type: text/plain, Size: 2728 bytes --]

(defun scrollup-n (&optional n)
  "Scroll the text up n (default 1) lines."
  (interactive "p")
  (scroll-up (or n 1))
)
(global-set-key [S-down] 'scrollup-n)

(defun scrolldown-n (&optional n)
  "Scroll the text down n (default 1) lines."
  (interactive "p")
  (scroll-down (or n 1))
)
(global-set-key [S-up] 'scrolldown-n)

(defun scrollup-6n (&optional n)
  "Scroll the text up 6n (default 6) lines."
  (interactive "p")
  (scroll-up (* 6 (or n 1)))
)
(global-set-key [C-S-down] 'scrollup-6n)

(defun scrolldown-6n (&optional n)
  "Scroll the text down 6n (default 6) lines."
  (interactive "p")
  (scroll-down (* 6 (or n 1)))
)
(global-set-key [C-S-up] 'scrolldown-6n)

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

(defun screen-top ()
  "Move the point to the top of the screen."
  (interactive)
  (move-to-window-line 0)
)
(global-set-key [C-left] 'screen-top)

(defun screen-bottom ()
  "Move the point to the bottom of the screen."
  (interactive)
  (move-to-window-line -1)
)
(global-set-key [C-right] 'screen-bottom)

(defun scroll-to-top (&optional n)
  "Scroll the current line to the top of the window"
  (interactive "P")
  (recenter (if n (prefix-numeric-value n) 0)))
(global-set-key [C-S-right] 'scroll-to-top)

(defun scroll-to-bottom (&optional n)
  "Scroll the current line to the bottom of the window.
If given a numerical arg, leave point that many lines from the bottom."
  (interactive "p")
  (recenter (- (or n 1))))
(global-set-key [C-S-left] 'scroll-to-bottom)

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

(defun 4-column ()
  "Disply the current buffer in 3 columns with follow-mode"
  (interactive)
  (follow-mode 1)
  (delete-other-windows)
  (split-window-horizontally) (split-window-horizontally)
  (split-window-horizontally)
  (balance-windows))
(global-set-key "\C-c4" '4-column)

(defun 3-column ()
  "Disply the current buffer in 3 columns with follow-mode"
  (interactive)
  (follow-mode 1)
  (delete-other-windows)
  (split-window-horizontally) (split-window-horizontally)
  (balance-windows))
(global-set-key "\C-c3" '3-column)

(defun 2-column ()
  "Disply the current buffer in 2 columns with follow-mode"
  (interactive)
  (follow-mode 1)
  (delete-other-windows)
  (split-window-horizontally)
  (balance-windows))
(global-set-key "\C-c2" '2-column)

(global-set-key "\C-c0" (lambda () (interactive) (message "follow-mode disabled") (follow-mode 0)))

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

(defun do-vm (arg)
  "Do (vertical-motion ARG), where ARG is a number, default of 0."
  (interactive "P")
  (vertical-motion (if arg (prefix-numeric-value arg) 0)))
(global-set-key "\C-cm" 'do-vm)

[-- Attachment #3: fragment.el --]
[-- Type: text/plain, Size: 11079 bytes --]

(tRER (fix-re--do-R*ER*-transform (ast-a "R*E*R+") 'car 0 0 1) (ast-a "R*\\(?:E+R+\\|R\\)"))
(tRER (fix-re--do-R*ER*-transform (ast-a "R*E*R*") 'car 0 0 1) (ast-a "R*\\(?:E+R*\\)?"))
(tRER (fix-re--do-R*ER*-transform (ast-a "R*E*F*R+") 'car 0 1 2) (ast-a "R*\\(?:\\(?:E*F+\\|E+\\)R+\\|R\\)"))
(tRER (fix-re--do-R*ER*-transform (ast-a "R*E*F*R*") 'car 0 1 2) (ast-a "R*\\(?:\\(?:E*F+\\|E+\\)R*\\)?"))
(tRER (fix-re--do-R*ER*-transform (ast-a "R*E\\{0\\}R+") 'car 0 0 1) (ast-a "R+"))
(tRER (fix-re--do-R*ER*-transform (ast-a "R*E\\{0\\}F\\{0\\}R*") 'car 0 1 2) (ast-a "R*"))

(tRER (fix-re--do-R*ER*-transform (ast-a "AR*E*R+") 'cadr 1 1 2) (ast-a "AR*\\(?:E+R+\\|R\\)"))
(tRER (fix-re--do-R*ER*-transform (ast-a "AR*E*R*") 'cadr 1 1 2) (ast-a "AR*\\(?:E+R*\\)?"))
(tRER (fix-re--do-R*ER*-transform (ast-a "AR*E*F*R+") 'cadr 1 2 3) (ast-a "AR*\\(?:\\(?:E*F+\\|E+\\)R+\\|R\\)"))
(tRER (fix-re--do-R*ER*-transform (ast-a "AR*E*F*R*") 'cadr 1 2 3) (ast-a "AR*\\(?:\\(?:E*F+\\|E+\\)R*\\)?"))
(tRER (fix-re--do-R*ER*-transform (ast-a "AR*E\\{0\\}R+") 'cadr 1 1 2) (ast-a "AR+"))
(tRER (fix-re--do-R*ER*-transform (ast-a "AR*E\\{0\\}F\\{0\\}R*") 'cadr 1 2 3) (ast-a "AR*"))

(tRER (fix-re--do-R*ER*-transform (ast-a "R*E*R+?") 'car 0 0 1) (ast-a "R*\\(?:E+R+?\\|R\\)"))
(tRER (fix-re--do-R*ER*-transform (ast-a "R*E*R*?") 'car 0 0 1) (ast-a "R*\\(?:E+R*?\\)??"))
(tRER (fix-re--do-R*ER*-transform (ast-a "R*E*F*R+?") 'car 0 1 2) (ast-a "R*\\(?:\\(?:E*F+\\|E+\\)R+?\\|R\\)"))
(tRER (fix-re--do-R*ER*-transform (ast-a "R*E*F*R*?") 'car 0 1 2) (ast-a "R*\\(?:\\(?:E*F+\\|E+\\)R*?\\)??"))
(tRER (fix-re--do-R*ER*-transform (ast-a "R*E\\{0\\}R+?") 'car 0 0 1) (ast-a "R+"))
(tRER (fix-re--do-R*ER*-transform (ast-a "R*E\\{0\\}F\\{0\\}R*?") 'car 0 1 2) (ast-a "R*"))

(tRER (fix-re--do-R*ER*-transform (ast-a "R+?E*R+") 'car 0 0 1) (ast-a "\\(?:R+?E+\\|R\\)R+"))
(tRER (fix-re--do-R*ER*-transform (ast-a "R*?E*R*") 'car 0 0 1) (ast-a "\\(?:R*?E+\\)??R*"))
(tRER (fix-re--do-R*ER*-transform (ast-a "R+?E*F*R+") 'car 0 1 2) (ast-a "\\(?:R+?\\(?:E*F+\\|E+\\)\\|R\\)R+\\)"))
(tRER (fix-re--do-R*ER*-transform (ast-a "R*?E*F*R*") 'car 0 1 2) (ast-a "\\(?:R*?\\(?:E*F+\\|E+\\)\\)??R*"))
(tRER (fix-re--do-R*ER*-transform (ast-a "R+?E\\{0\\}R+") 'car 0 0 1) (ast-a "RR+"))
(tRER (fix-re--do-R*ER*-transform (ast-a "R*?E\\{0\\}F\\{0\\}R*") 'car 0 1 2) (ast-a "R*"))

(tRER (fix-re--do-R*ER*-transform (ast-a "R*?E*R+?") 'car 0 0 1) (ast-a "R*?\\(?:E+R+?\\|R\\)"))
(tRER (fix-re--do-R*ER*-transform (ast-a "R*?E*R*?") 'car 0 0 1) (ast-a "R*?\\(?:E+R*?\\)??"))
(tRER (fix-re--do-R*ER*-transform (ast-a "R*?E*F*R+?") 'car 0 1 2) (ast-a "R*?\\(?:\\(?:E*F+\\|E+\\)R+?\\|R\\)"))
(tRER (fix-re--do-R*ER*-transform (ast-a "R*?E*F*R*?") 'car 0 1 2) (ast-a "R*?\\(?:\\(?:E*F+\\|E+\\)R*?\\)??"))
(tRER (fix-re--do-R*ER*-transform (ast-a "R*?E\\{0\\}R+?") 'car 0 0 1) (ast-a "R+?"))
(tRER (fix-re--do-R*ER*-transform (ast-a "R*?E\\{0\\}F\\{0\\}R*?") 'car 0 1 2) (ast-a "R*?"))

(defun fix-re--R+ER*->R+\(E@R*\)\? (ptr ad)
  "Do R+ER* -> R+(E@R*)? or R+ER+ -> R+(E@R+|R) on the whole list.
PTR/AD point at the first element of the sequential list.

Here, E is a non-empty sequence of elements which are matched by
the empty string, E@ is the \"de-emptified\" version of E."
  ;; We must perform the loop rightmost transformations first.  To see this,
  ;; consider R*ER*FR* done leftmost first.  The first transformation takes us
  ;; to R*(E@R*)?FR*.  We're now stuck, as the middle R* is no longer
  ;; "exposed" to the last R*, and the end expression is still ill-formed.
  ;; Done rightmost first, R*ER*FR* -> R*ER*(F@R*)? -> R*(E@R*)?(F@R*)?, which
  ;; is well-formed.
  (let (res)
    (let ((ptr ptr) (ad ad))
      (when (fix-re--ptr-next ptr ad)
	(setq res (fix-re--R+ER*->R+\(E@R*\)\? ptr ad))))

    (let* ((elt-ptr ptr)
	   (elt-ad ad)
	   (elt (fix-re--ptr-get elt-ptr elt-ad))
	   R0-R
	   empty0-ptr empty1-ptr)	; No need for ..-ad's, since
					; these will always be 'cadr.
      (or
      ;; Is `elt' R+ or R*?
       (when (and (consp elt)
		  (memq (car elt) '(+ *)))
	 (setq R0-R (cdr elt))
	 ;; Is the next element one matching the empty string, and which
	 ;; isn't R+ or R*?
	 (setq elt (fix-re--ptr-next elt-ptr elt-ad))
	 (when (and elt
		    (fix-re--matches-empty-p elt)
		    (not (and (consp elt)
			      (memq (car elt) '(+ *))
			      (equal (cdr elt) R0-R))))
	   (setq empty0-ptr elt-ptr ; Remember first empty. -ad is implicitly 'cadr
		 empty1-ptr elt-ptr )	; Remember last empty.
	   ;; Read the elements which match empty, but aren't R+ or R*.
	   (while (and (setq elt (fix-re--ptr-next elt-ptr elt-ad))
		       (fix-re--matches-empty-p elt)
		       (not (and (consp elt)
				 (memq (car elt) '(+ *))
				 (equal (cdr elt) R0-R))))
	     (setq empty1-ptr elt-ptr))
	   ;; Have we found the matching R+ or R*?
	   (when (and elt
		      (consp elt)
		      (memq (car elt) '(+ *))
		      (equal (cdr elt) R0-R))
	     ;; Yes.  We're in business.
	     (fix-re--do-R*ER*-transform ptr ad empty0-ptr empty1-ptr elt-ptr)
	     t)))
       res))))

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; R*(R*A|B) -> R*(A|B)
(defun fix-re--\[R\]+\(\[S\]*\)-transform (ptr ad R*)
  "Attempt to transform an alternative which begins with [..]+.
The entire construct we have looks like [R]*([S]*A|...), where CA1 is the
\"outside\" char-alt, and PTR/AD points at the entire alternative.
The *s on the char-alts may alternatively be +s.

In the following \"[R-S]\" denotes the difference of the two character
alteratives R and S.  [R-S] is a new character alternative which matches any
character which R does, but S doesn't.

The transformation looks like:
\(i) (with ^ operators)
    [^R]+([^S]*A|...) -> [^R]+(([R-S][^S]*)?A|...)
    [^R]+([^S]+A|...) -> [^R]+(([R-S][^S]*)?[^S]A|...)

\(ii) (without ^ operators)
    [R]+([S]*A|...) -> [R]+(([S-R][S]*)?A|...)   or
    [R]+([S]+A|...) -> [R]+(([S-R][S]*)?[S]A|...)
or, if S-R is empty:
    [R]+([S]*A|...) -> [R]+(A|...)
    [R]+([S]+A|...) -> [R]+([S]A|...)

FIXME!!!
"
  (let* ((elt (fix-re--ptr-get ptr ad))	; [S]*A,,,
	 (S* (car elt))			; [S]*
	 (S (cdr S*))			; [S]
	 (R (cdr R*))			; [R]
	 S-R R-S new-S new-\(
	 res subres
	)
    (when (eq (cadr S) (cadr R)) ; Either both have or neither has ^ operator.
      (when			 ; Have R and S got any overlap?
	  (if (cadr S)		 ; With ^ operator.
	      (progn
		(setq R-S (copy-tree R))
		(setcar (cdr R-S) nil)	; Remove the ^.
		(setq subres (fix-re--chalt-minus (cdr R-S) (caddr S) t)))
	    ;; Transform [abc]+([cde]*R|...) to [abc]+(([de][cde]*)?R|...)
	    (setq S-R (copy-tree S))
	    (setq subres (fix-re--chalt-minus (cdr S-R) (caddr R) nil)))
	(if (null (caddr (or S-R R-S)))
	    (progn
	      (if (eq (car S*) '+)
		  (fix-re--chop-+* elt 'car) ; [abc]+([ab]+A|..) -> [abc]+([ab]A|..)
		(fix-re--ind-chop ptr ad 'car t)) ; [abc]+([ab]*A|..) -> [abc]+(A|..)
	      (setq res t))
	  (fix-re--wrap-in-\( '\\\(\?: elt 'car)
	  (setq new-\( (fix-re--ptr-get elt 'car))
	  (fix-re--insert (if (numberp subres) subres (or S-R R-S))
			  (cadr new-\() 'car)
	  (fix-re--+*ify '\? elt 'car)
	  (when (eq (car S*) '+)
	    (setcar S* '*)
	    (setq new-S (copy-tree S))
	    (fix-re--insert-after new-S elt 'car))
	  (setq res t))))
    res))
	  
(tp (fix-re--\[R\]+\(\[S\]*\)-transform (ast-aa "\\([cde]*G\\)") 'cadr (ast-aa "[abc]+")) 1 (ast-aa "\\(\\(?:[de][cde]*\\)?G\\)") t t)
(tp (fix-re--\[R\]+\(\[S\]*\)-transform (ast-aa "\\([cde]+G\\)") 'cadr (ast-aa "[abc]+")) 1 (ast-aa "\\(\\(?:[de][cde]*\\)?[cde]G\\)") t t)
(tp (fix-re--\[R\]+\(\[S\]*\)-transform (ast-aa "\\([ab]*G\\)") 'cadr (ast-aa "[abc]+")) 1 (ast-aa "\\(G\\)") t t)
(tp (fix-re--\[R\]+\(\[S\]*\)-transform (ast-aa "\\([ab]+G\\)") 'cadr (ast-aa "[abc]+")) 1 (ast-aa "\\([ab]G\\)") t t)

(tp (fix-re--\[R\]+\(\[S\]*\)-transform (ast-aa "\\([^cde]*G\\)") 'cadr (ast-aa "[^abc]+")) 1 (ast-aa "\\(\\(?:[ab][^cde]*\\)?G\\)") t t)
(tp (fix-re--\[R\]+\(\[S\]*\)-transform (ast-aa "\\([^cde]+G\\)") 'cadr (ast-aa "[^abc]+")) 1 (ast-aa "\\(\\(?:[ab][^cde]*\\)?[^cde]G\\)") t t)
(tp (fix-re--\[R\]+\(\[S\]*\)-transform (ast-aa "\\([^abc]*G\\)") 'cadr (ast-aa "[^ab]+")) 1 (ast-aa "\\(G\\)") t t)
(tp (fix-re--\[R\]+\(\[S\]*\)-transform (ast-aa "\\([^abc]+G\\)") 'cadr (ast-aa "[^ab]+")) 1 (ast-aa "\\([^abc]G\\)") t t)

(tp (fix-re--\[R\]+\(\[S\]*\)-transform (ast-aa "\\([cde]*G\\)") 'cadr (ast-aa "[^abc]")) 1 (ast-aa "\\([cde]*G\\)") 'nil t) (tp (fix-re--\[R\]+\(\[S\]*\)-transform (ast-aa "\\([^cde]*G\\)") 'cadr (ast-aa "[abc]")) 1 (ast-aa "\\([^cde]*G\\)") 'nil t)
(tp (fix-re--\[R\]+\(\[S\]*\)-transform (ast-aa "\\([def]*G\\)") 'cadr (ast-aa "[abc]")) 1 (ast-aa "\\([def]*G\\)") 'nil t)

(defun fix-re--do-R+\(R*A|B\)-transform (R-rep alt)
  "Attempt a R+(R*A|B) -> R+(A|B) transformation.
R-REP is a cons representing either R+ or R*.  ALT represents a
form of the form \(..\|..\|...\).
"
  (let* ((R-R (cdr R-rep))
	 (ptr alt)
	 (ad 'cadr) ; Point to the second elt. of the list, the first being '\\\(
	 (elt (fix-re--ptr-get ptr ad))
	 res car-elt elt-+*)
    (while elt				  ; (R*A)
      (when (and (consp elt)		  ; This should always be true
		 (setq car-elt (car elt)) ; This is now R*A
		 (consp car-elt)
		 (memq (setq elt-+* (car car-elt)) '(+ *)))
	(cond
	 ((equal (cdr car-elt) R-R)
	  (if (eq elt-+* '+)
	      (fix-re--chop-+* elt 'car)
	    (fix-re--ind-chop ptr ad 'car t)) ; i.e. ~ (fix-re--chop elt 'car)
	  (setq res t))
	 ((and (consp R-R) (eq (car R-R) '\[)
	       (consp (cdr car-elt)) (eq (cadr car-elt) '\[))
	  (if (fix-re--\[R\]+\(\[S\]*\)-transform ptr ad R-rep)
	      (setq res t)))))
      (setq elt (fix-re--ptr-next ptr ad)))
    res))

(defun fix-re--R+\(R*A|B\)->R*\(A|B\) (ptr ad)
  "Do the transition on every pertinent element pairs in the sequence.
PTR/AD point to the first element in the sequential list."
  (let ((elt (fix-re--ptr-get ptr ad))
	R-rep res)
    (while elt
      (if (and (consp elt)
	       (memq (car elt) '(+ *)))
	  (progn
	    (setq R-rep elt
		  elt (fix-re--ptr-next ptr ad))
	    (when (fix-re--is-\( elt)
	      (if (fix-re--do-R+\(R*A|B\)-transform R-rep elt)
		  (setq res t))
	      (setq elt (fix-re--ptr-next ptr ad))))
	(setq elt (fix-re--ptr-next ptr ad))))
    res))

\f
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;;
(defun fix-re--\(R|R\)->\(R\) (ptr ad)
  "Remove duplicate elements from the alternatives list.
PTR/AD point to the first element of the list, which will be a symbol like
'\\\(."
  (let* ((elt0-ptr ptr) (elt0-ad ad)
	 (elt0 (fix-re--ptr-next elt0-ptr elt0-ad))
	 elt1-ptr elt1-ad elt1
	 res)
    (while elt0
      (setq elt1-ptr elt0-ptr
	    elt1-ad elt0-ad
	    elt1 (fix-re--ptr-next elt1-ptr elt1-ad))
      (while elt1
	(while (equal elt1 elt0)
	  (fix-re--chop elt1-ptr elt1-ad)
	  (setq elt1 (fix-re--ptr-get elt1-ptr elt1-ad))
	  (setq res t))
	(setq elt1 (fix-re--ptr-next elt1-ptr elt1-ad)))
      (setq elt0 (fix-re--ptr-next elt0-ptr elt0-ad)))
    res))

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

Re: "... the window start at a meaningless point within a line."

[Eli Zaretskii]

> Date: Wed, 28 Oct 2015 13:15:05 +0000
> Cc: emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> After applying the patch, to test things, in X (?or windows)
> o - Start emacs -Q.
> o - Load utilities.el.
> o - Visit fragment.el.
> o - Using the mouse, make the frame wide enough for two side by side
>   windows of width 79 and 80.
> o - C-x 3.
> o - Fiddle about with the mouse, M-{, M-}, until the windows are 79, and
>   80 wide (`window-body-width' is your friend, here).
> o - Scroll the buffer until L162 straddles window 1 and window 2, with
>   two display lines in each window.  (S-<up>/<down>, C-S-<up>/<down>
>   from utilies.el are handy, here).
> 
> First of all, note that in L162, all characters are displayed exactly
> once.  This is the purpose of this change.
> 
> With point in W2, note that C-<left> doesn't go to the beginning of
> the window, and C-<right> doesn't go to the end.  The actual end
> positions are a line out in both cases.  With point in the split line,
> note that vertical-motion (C-u <n> C-c m) doesn't go to the right
> places, even when the target line is below the split line.  Typing <up>
> and <down> also produce interesting effects.
> 
> Using M-{ or M-}, make W1 80 wide, and W2 79 wide.  Scroll the windows
> up and down, back to the same place, to ensure that Follow Mode has
> resynchronised its windows[*].  Now repeat the experiments of the
> previous paragraph.  If anything, the results now are even worse.

The patch below should go a long way towards solving these problems.
The only issue I left unsolved is exposed when typing C-p on the
second screen line of the right-side window: you will see that it
doesn't succeed in keeping the visual column.  Solving this is left as
an exercise ;-)

I hope the changes I made and the comments to those changes will speak
for themselves, and show how to solve such problems.

Please note that there's a very serious underlying problem here: the
move_it_* functions and all their callers assume that moving beyond
above or below the window keeps the same window dimensions, in
particularly its width.  Follow mode violates this assumption, because
moving from the left-side to the right-side window or vice versa can
change the width.  Accounting for this would need very deep changes in
the above mentioned functions.  For now, I avoided that, and the patch
below is just a band-aid.  If it works well enough in practice, maybe
we can avoid the surgery.  But in general, I must say that fixing what
you are trying to fix is an ambitious undertaking; perhaps it would be
better to make sure the two windows are always the same width, using
pixel-wise resizing where necessary.

diff --git a/src/indent.c b/src/indent.c
index 32597bb..993ef5f 100644
--- a/src/indent.c
+++ b/src/indent.c
@@ -2034,6 +2034,7 @@ whether or not it is currently displayed in some window.  */)
   else
     {
       ptrdiff_t it_start, it_overshoot_count = 0;
+      ptrdiff_t w_start = marker_position (w->start);
       int first_x;
       bool overshoot_handled = 0;
       bool disp_string_at_start_p = 0;
@@ -2043,6 +2044,8 @@ whether or not it is currently displayed in some window.  */)
       int start_x IF_LINT (= 0);
       int to_x = -1;
       struct text_pos xdisp_ws;
+      bool on_window_border = false;
+      ptrdiff_t goal_pt = PT;
 
       bool start_x_given = !NILP (cur_col);
 
@@ -2102,27 +2105,40 @@ whether or not it is currently displayed in some window.  */)
 	     do this, we start moving with IT->current_x == 0, while PT is
 	     really at some x > 0.  */
 	  reseat_at_previous_visible_line_start (&it);
+	  /* If we are in a right-side window under follow-mode, and
+	     previous visible line start is before that window's
+	     start, reseat to the window's start point instead, where
+	     we know the horizontal coordinates are zero.  This avoids
+	     problems in iterator metrics when the 2 windows under
+	     follow-mode have different widths.  */
+	  if (w->exact_start
+	      && IT_CHARPOS (it) < w_start)
+	    {
+	      on_window_border = true;
+	      reseat_at_window_start (&it);
+	    }
 	  it.current_x = it.hpos = 0;
 	}
       if (IT_CHARPOS (it) != PT)
-	/* We used to temporarily disable selective display here; the
-	   comment said this is "so we don't move too far" (2005-01-19
-	   checkin by kfs).  But this does nothing useful that I can
-	   tell, and it causes Bug#2694 .  -- cyd */
-	/* When the position we started from is covered by a display
-	   string, move_it_to will overshoot it, while vertical-motion
-	   wants to put the cursor _before_ the display string.  So in
-	   that case, we move to buffer position before the display
-	   string, and avoid overshooting.  But if the position before
-	   the display string is a newline, we don't do this, because
-	   otherwise we will end up in a screen line that is one too
-	   far back.  */
-	move_it_to (&it,
-		    (!disp_string_at_start_p
-		     || FETCH_BYTE (IT_BYTEPOS (it)) == '\n')
-		    ? PT
-		    : PT - 1,
-		    -1, -1, -1, MOVE_TO_POS);
+	{
+	  /* We used to temporarily disable selective display here; the
+	     comment said this is "so we don't move too far" (2005-01-19
+	     checkin by kfs).  But this does nothing useful that I can
+	     tell, and it causes Bug#2694 .  -- cyd */
+	  /* When the position we started from is covered by a display
+	     string, move_it_to will overshoot it, while vertical-motion
+	     wants to put the cursor _before_ the display string.  So in
+	     that case, we move to buffer position before the display
+	     string, and avoid overshooting.  But if the position before
+	     the display string is a newline, we don't do this, because
+	     otherwise we will end up in a screen line that is one too
+	     far back.  */
+	  goal_pt =
+	    (!disp_string_at_start_p || FETCH_BYTE (IT_BYTEPOS (it)) == '\n')
+	    ? PT
+	    : PT - 1;
+	  move_it_to (&it, goal_pt, -1, -1, -1, MOVE_TO_POS);
+	}
 
       /* IT may move too far if truncate-lines is on and PT lies
 	 beyond the right margin.  IT may also move too far if the
@@ -2176,7 +2192,9 @@ whether or not it is currently displayed in some window.  */)
 	  /* Do this even if LINES is 0, so that we move back to the
 	     beginning of the current line as we ought.  */
 	  if ((nlines < 0 && IT_CHARPOS (it) > 0)
-	      || (nlines == 0 && !(start_x_given && start_x <= to_x)))
+	      || (nlines == 0
+		  && !(start_x_given && start_x <= to_x)
+		  && !on_window_border))
 	    move_it_by_lines (&it, max (PTRDIFF_MIN, nlines));
 	}
       else if (overshoot_handled)
@@ -2212,6 +2230,35 @@ whether or not it is currently displayed in some window.  */)
 	    }
 	}
 
+      /* If we moved up in the right-side window under follow-mode,
+	 and found ourselves beyond the window's start, we must move
+	 back from window start to PT, see how many lines is that,
+	 then switch to the left-side window, and move back the rest
+	 of NLINES starting from that window's last line.  That's
+	 because move_it_* functions use the window dimensions, which
+	 might be different in the other window.  */
+      if (w->exact_start && IT_CHARPOS (it) < w_start)
+	{
+	  int vpos = it.vpos, nlines_prev_window;
+	  Lisp_Object prev_window = w->prev;
+	  struct window *pw;
+	  ptrdiff_t prev_window_endpos;
+	  struct text_pos pw_top;
+
+	  eassert (WINDOW_LIVE_P (prev_window));
+	  reseat_at_window_start (&it);
+	  it.current_x = it.hpos = it.vpos = 0;
+	  move_it_to (&it, goal_pt, -1, -1, -1, MOVE_TO_POS);
+	  nlines_prev_window = nlines + it.vpos + 1;
+	  pw = XWINDOW (prev_window);
+	  SET_TEXT_POS_FROM_MARKER (pw_top, pw->start);
+	  start_display (&it, pw, pw_top);
+	  move_it_to (&it, -1, -1, it.last_visible_y - 1, -1, MOVE_TO_Y);
+	  move_it_by_lines (&it, nlines_prev_window);
+	  /* Pretend the original move worked as intended.  */
+	  it.vpos = vpos;
+	}
+
       /* Move to the goal column, if one was specified.  If the window
 	 was originally hscrolled, the goal column is interpreted as
 	 an addition to the hscroll amount.  */
diff --git a/src/window.c b/src/window.c
index b922511..38aadd6 100644
--- a/src/window.c
+++ b/src/window.c
@@ -7149,7 +7149,7 @@ and scrolling positions.  */)
 
 DEFUN ("window-test-dump", Fwindow_test_dump, Swindow_test_dump, 0, 0, "",
        doc: /* Dump some critical components of the selected window to `message'.*/)
-  ()
+  (void)
 {
   Lisp_Object window = Fselected_window ();
   struct window *w = decode_live_window (window);

Re: "... the window start at a meaningless point within a line."

[Alan Mackenzie]

Hello, Eli,

Thanks for this.

On Sat, Oct 31, 2015 at 03:21:02PM +0200, Eli Zaretskii wrote:
> > Date: Wed, 28 Oct 2015 13:15:05 +0000
> > Cc: emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

[ .... ]

> The patch below should go a long way towards solving these problems.
> The only issue I left unsolved is exposed when typing C-p on the
> second screen line of the right-side window: you will see that it
> doesn't succeed in keeping the visual column.  Solving this is left as
> an exercise ;-)

> I hope the changes I made and the comments to those changes will speak
> for themselves, and show how to solve such problems.

> Please note that there's a very serious underlying problem here: the
> move_it_* functions and all their callers assume that moving beyond
> above or below the window keeps the same window dimensions, in
> particularly its width.  Follow mode violates this assumption, because
> moving from the left-side to the right-side window or vice versa can
> change the width.  Accounting for this would need very deep changes in
> the above mentioned functions.  For now, I avoided that, and the patch
> below is just a band-aid.  If it works well enough in practice, maybe
> we can avoid the surgery.  But in general, I must say that fixing what
> you are trying to fix is an ambitious undertaking; perhaps it would be
> better to make sure the two windows are always the same width, using
> pixel-wise resizing where necessary.

I'm aware that it's a difficult thing to achieve.  But pixel-wise
resizing isn't a good idea on ttys.  The only "equivalent" there would
be to thicken borders between windows to make them equal width.  I can't
see that being a popular move.

Anyhow, I played around with the new indent.c a bit, got a seg fault,
rebuilt it with debugging, and reproduced the said seg fault.

At the time, I had Follow Mode active on a buffer in three side by side
windows in the initial frame.  Point was in the left window, and I
attempted to scroll the window by either a single line, or six lines
(the latter, I think).  The segfault happened here:
 
> +      /* If we moved up in the right-side window under follow-mode,
> +	 and found ourselves beyond the window's start, we must move
> +	 back from window start to PT, see how many lines is that,
> +	 then switch to the left-side window, and move back the rest
> +	 of NLINES starting from that window's last line.  That's
> +	 because move_it_* functions use the window dimensions, which
> +	 might be different in the other window.  */
> +      if (w->exact_start && IT_CHARPOS (it) < w_start)
> +	{
> +	  int vpos = it.vpos, nlines_prev_window;
> +	  Lisp_Object prev_window = w->prev;
> +	  struct window *pw;
> +	  ptrdiff_t prev_window_endpos;
> +	  struct text_pos pw_top;
> +
> +	  eassert (WINDOW_LIVE_P (prev_window));
> +	  reseat_at_window_start (&it);
> +	  it.current_x = it.hpos = it.vpos = 0;
> +	  move_it_to (&it, goal_pt, -1, -1, -1, MOVE_TO_POS);
> +	  nlines_prev_window = nlines + it.vpos + 1;
> +	  pw = XWINDOW (prev_window);
> +	  SET_TEXT_POS_FROM_MARKER (pw_top, pw->start);   <=================================
> +	  start_display (&it, pw, pw_top);
> +	  move_it_to (&it, -1, -1, it.last_visible_y - 1, -1, MOVE_TO_Y);
> +	  move_it_by_lines (&it, nlines_prev_window);
> +	  /* Pretend the original move worked as intended.  */
> +	  it.vpos = vpos;
> +	}
> +
>        /* Move to the goal column, if one was specified.  If the window
>  	 was originally hscrolled, the goal column is interpreted as
>  	 an addition to the hscroll amount.  */

, and it happened because prev_window is 0.  Presumably the left hand
window on the first frame has no previous.  I can't see how the
WINDOW_LIVE_P check failed to catch this.

I haven't spent enough time studying the patch yet to be able to
understand it.

Anyhow, just in case the cause is obvious, here's the relevant section
from my gdb session:

Program received signal SIGSEGV, Segmentation fault.
0x00000000005b150f in Fvertical_motion (lines=-2, window=13422013, cur_col=0) at indent.c:2254
2254              SET_TEXT_POS_FROM_MARKER (pw_top, pw->start);
(gdb) p pw->start
Cannot access memory at address 0x53
(gdb) p pw
$1 = (struct window *) 0xfffffffffffffffb
(gdb) p window->prev
Attempt to extract a component of a value that is not a structure pointer.
(gdb) p w->prev
$2 = 0
(gdb) p prev_window
$3 = 0
(gdb) p w->prev
$4 = 0
(gdb) p WINDOW_LIVE_P (prev_window)
$5 = 0
(gdb) bt
#0  0x00000000005b150f in Fvertical_motion (lines=-2, window=13422013, cur_col=0) at indent.c:2254
#1  0x00000000004a7016 in window_scroll_line_based (window=13422013, n=-1, whole=false, noerror=false) at window.c:5206
#2  0x00000000004a56df in window_scroll (window=13422013, n=-1, whole=false, noerror=false) at window.c:4726
#3  0x00000000004a7654 in scroll_command (n=6, direction=-1) at window.c:5341
#4  0x00000000004a76b8 in Fscroll_down (arg=6) at window.c:5369
#5  0x00000000005f856b in Ffuncall (nargs=2, args=0x7ffca6e8f8b8) at eval.c:2650
#6  0x000000000063da3b in exec_byte_code (bytestr=45853268, vector=16338405, maxdepth=10, args_template=0, nargs=0, args=0x0) at bytecode.c:880
#7  0x00000000005f90df in funcall_lambda (fun=16338501, nargs=1, arg_vector=0xf94de5) at eval.c:2876
#8  0x00000000005f8b7d in apply_lambda (fun=16338501, args=17625955, count=9) at eval.c:2751
#9  0x00000000005f72fb in eval_sub (form=17940163) at eval.c:2168
#10 0x00000000005f3542 in Fprogn (body=21580531) at eval.c:427
#11 0x00000000005f904e in funcall_lambda (fun=21580291, nargs=1, arg_vector=0x7ffca6e90160) at eval.c:2869
#12 0x00000000005f88f4 in Ffuncall (nargs=2, args=0x7ffca6e90158) at eval.c:2711
#13 0x00000000005f041c in Ffuncall_interactively (nargs=2, args=0x7ffca6e90158) at callint.c:248
#14 0x00000000005f845a in Ffuncall (nargs=3, args=0x7ffca6e90150) at eval.c:2630
#15 0x00000000005f2aba in Fcall_interactively (function=3125504, record_flag=0, keys=13390261) at callint.c:836
#16 0x00000000005f85e5 in Ffuncall (nargs=4, args=0x7ffca6e90488) at eval.c:2657
#17 0x000000000063da3b in exec_byte_code (bytestr=10398516, vector=10398549, maxdepth=54, args_template=4102, nargs=1, args=0x7ffca6e909f0) at bytecode.c:880
#18 0x00000000005f8daf in funcall_lambda (fun=10398469, nargs=1, arg_vector=0x7ffca6e909e8) at eval.c:2810
#19 0x00000000005f881d in Ffuncall (nargs=2, args=0x7ffca6e909e0) at eval.c:2699
#20 0x00000000005f7f5e in call1 (fn=14736, arg1=3125504) at eval.c:2509
#21 0x0000000000559da8 in command_loop_1 () at keyboard.c:1458
#22 0x00000000005f5279 in internal_condition_case (bfun=0x55959c <command_loop_1>, handlers=18912, hfun=0x558d60 <cmd_error>) at eval.c:1309
#23 0x00000000005592a4 in command_loop_2 (ignore=0) at keyboard.c:1086
#24 0x00000000005f4a6e in internal_catch (tag=45552, func=0x55927b <command_loop_2>, arg=0) at eval.c:1073
#25 0x0000000000559244 in command_loop () at keyboard.c:1065
#26 0x0000000000558935 in recursive_edit_1 () at keyboard.c:671
#27 0x0000000000558ac2 in Frecursive_edit () at keyboard.c:742
#28 0x00000000005568b2 in main (argc=2, argv=0x7ffca6e90e88) at emacs.c:1644
(gdb)


-- 
Alan Mackenzie (Nuremberg, Germany).

Re: "... the window start at a meaningless point within a line."

[Eli Zaretskii]

> Date: Sat, 31 Oct 2015 21:17:40 +0000
> Cc: emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> pixel-wise resizing isn't a good idea on ttys.

It isn't supposed to be needed there.

> > +	  eassert (WINDOW_LIVE_P (prev_window));
> > +	  reseat_at_window_start (&it);
> > +	  it.current_x = it.hpos = it.vpos = 0;
> > +	  move_it_to (&it, goal_pt, -1, -1, -1, MOVE_TO_POS);
> > +	  nlines_prev_window = nlines + it.vpos + 1;
> > +	  pw = XWINDOW (prev_window);
> > +	  SET_TEXT_POS_FROM_MARKER (pw_top, pw->start);   <=================================
> > +	  start_display (&it, pw, pw_top);
> > +	  move_it_to (&it, -1, -1, it.last_visible_y - 1, -1, MOVE_TO_Y);
> > +	  move_it_by_lines (&it, nlines_prev_window);
> > +	  /* Pretend the original move worked as intended.  */
> > +	  it.vpos = vpos;
> > +	}
> > +
> >        /* Move to the goal column, if one was specified.  If the window
> >  	 was originally hscrolled, the goal column is interpreted as
> >  	 an addition to the hscroll amount.  */
> 
> , and it happened because prev_window is 0.  Presumably the left hand
> window on the first frame has no previous.  I can't see how the
> WINDOW_LIVE_P check failed to catch this.

It is imperative to understand why that test missed, maybe you didn't
compile with --enable-checking?

Anyway, if this can legitimately happen, i.e. if the exact_start flag
can be set in the left-side window of the two windows under Follow
mode, then the WINDOW_LIVE_P test should be in an 'if' test, and the
following code should be skipped if the test fails.

Re: "... the window start at a meaningless point within a line."

[Alan Mackenzie]

Hello, Eli.

On Sun, Nov 01, 2015 at 05:40:15AM +0200, Eli Zaretskii wrote:
> > Date: Sat, 31 Oct 2015 21:17:40 +0000
> > Cc: emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>
> > 
> > pixel-wise resizing isn't a good idea on ttys.

> It isn't supposed to be needed there.

> > > +	  eassert (WINDOW_LIVE_P (prev_window));
> > > +	  reseat_at_window_start (&it);
> > > +	  it.current_x = it.hpos = it.vpos = 0;
> > > +	  move_it_to (&it, goal_pt, -1, -1, -1, MOVE_TO_POS);
> > > +	  nlines_prev_window = nlines + it.vpos + 1;
> > > +	  pw = XWINDOW (prev_window);
> > > +	  SET_TEXT_POS_FROM_MARKER (pw_top, pw->start);   <=================================
> > > +	  start_display (&it, pw, pw_top);
> > > +	  move_it_to (&it, -1, -1, it.last_visible_y - 1, -1, MOVE_TO_Y);
> > > +	  move_it_by_lines (&it, nlines_prev_window);
> > > +	  /* Pretend the original move worked as intended.  */
> > > +	  it.vpos = vpos;
> > > +	}
> > > +
> > >        /* Move to the goal column, if one was specified.  If the window
> > >  	 was originally hscrolled, the goal column is interpreted as
> > >  	 an addition to the hscroll amount.  */
> > 
> > , and it happened because prev_window is 0.  Presumably the left hand
> > window on the first frame has no previous.  I can't see how the
> > WINDOW_LIVE_P check failed to catch this.

> It is imperative to understand why that test missed, maybe you didn't
> compile with --enable-checking?

I hadn't compiled with --enable-checking.

> Anyway, if this can legitimately happen, i.e. if the exact_start flag
> can be set in the left-side window of the two windows under Follow
> mode, then the WINDOW_LIVE_P test should be in an 'if' test, and the
> following code should be skipped if the test fails.

DONE.  This version now doesn't segfault.  So I'll go back to your
previous mail to make some more comments.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: "... the window start at a meaningless point within a line."

[Alan Mackenzie]

Hello, Eli.

On Sat, Oct 31, 2015 at 03:21:02PM +0200, Eli Zaretskii wrote:
> > Date: Wed, 28 Oct 2015 13:15:05 +0000
> > Cc: emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > After applying the patch, to test things, in X (?or windows)
> > o - Start emacs -Q.
> > o - Load utilities.el.
> > o - Visit fragment.el.
> > o - Using the mouse, make the frame wide enough for two side by side
> >   windows of width 79 and 80.
> > o - C-x 3.
> > o - Fiddle about with the mouse, M-{, M-}, until the windows are 79, and
> >   80 wide (`window-body-width' is your friend, here).
> > o - Scroll the buffer until L162 straddles window 1 and window 2, with
> >   two display lines in each window.  (S-<up>/<down>, C-S-<up>/<down>
> >   from utilies.el are handy, here).

> > First of all, note that in L162, all characters are displayed exactly
> > once.  This is the purpose of this change.

> > With point in W2, note that C-<left> doesn't go to the beginning of
> > the window, and C-<right> doesn't go to the end.  The actual end
> > positions are a line out in both cases.  With point in the split line,
> > note that vertical-motion (C-u <n> C-c m) doesn't go to the right
> > places, even when the target line is below the split line.  Typing <up>
> > and <down> also produce interesting effects.

> > Using M-{ or M-}, make W1 80 wide, and W2 79 wide.  Scroll the windows
> > up and down, back to the same place, to ensure that Follow Mode has
> > resynchronised its windows[*].  Now repeat the experiments of the
> > previous paragraph.  If anything, the results now are even worse.

> The patch below should go a long way towards solving these problems.
> The only issue I left unsolved is exposed when typing C-p on the
> second screen line of the right-side window: you will see that it
> doesn't succeed in keeping the visual column.  Solving this is left as
> an exercise ;-)

;-).  Sometimes it doesn't even go to the right line (i.e. when there
are more or fewer "actual" lines than "xdisp" lines).  To detect this
situation necessarily involves reseat_at_window_start followed by moving
forward, counting (display) line starts, in some fashion.  Do you agree?
To move to the right column of the line necessarily involves knowing
where the "actual" BOL is.  This also involves a reseat_at_window_start
and moving forward lines.  Do you agree this one, too?

At the moment, I can't see how post analysis (i.e. analysis after
Fvertical_motion has done move_it_by_lines (&it, nlines)) can be
avoided.

In this line of thinking, considering how bidi complicates the order of
PT, end position, and various BOLs, it might be very convenient to
enhance move_it_in_display_line_to to take TWO (or even several)
to_charpos parameters, such that the function would stop after reaching
either one of them.  (Naturally, there would be some convention for the
usual case of calling it with just one to_charpos.)  Although this
enhancement would be tedious, it would surely not be difficult.  With
such a function, we could scan through the buffer with both PT and
end_position as to_charpos's.  What do you think?

> I hope the changes I made and the comments to those changes will speak
> for themselves, and show how to solve such problems.

I like your idea for solving the "going back over the top of the window"
problem.  Is there any guarantee that w->prev will actually be the
window "to the left of" w (whatever that might mean)?

> Please note that there's a very serious underlying problem here: the
> move_it_* functions and all their callers assume that moving beyond
> above or below the window keeps the same window dimensions, in
> particularly its width.  Follow mode violates this assumption, because
> moving from the left-side to the right-side window or vice versa can
> change the width.  Accounting for this would need very deep changes in
> the above mentioned functions.  For now, I avoided that, and the patch
> below is just a band-aid.  If it works well enough in practice, maybe
> we can avoid the surgery.  But in general, I must say that fixing what
> you are trying to fix is an ambitious undertaking; perhaps it would be
> better to make sure the two windows are always the same width, using
> pixel-wise resizing where necessary.

As already said, this would be sub-optimal for ttys, where the total
width is fixed, and tends not to be a number of the form 2n+1 or 3n+2.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: "... the window start at a meaningless point within a line."

[Eli Zaretskii]

> Date: Sun, 1 Nov 2015 15:23:54 +0000
> Cc: emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > The patch below should go a long way towards solving these problems.
> > The only issue I left unsolved is exposed when typing C-p on the
> > second screen line of the right-side window: you will see that it
> > doesn't succeed in keeping the visual column.  Solving this is left as
> > an exercise ;-)
> 
> ;-).  Sometimes it doesn't even go to the right line (i.e. when there
> are more or fewer "actual" lines than "xdisp" lines).

The code is explicitly unfinished, I said that much.  I wrote it in
the hope that it will show enough to get you off the ground with using
the move_it_* functions for the purposes such as this.  But it
definitely needs more work.

> To detect this situation necessarily involves reseat_at_window_start
> followed by moving forward, counting (display) line starts, in some
> fashion.  Do you agree?

I don't know if I agree.  I don't think I understand the situation
well enough to reason about it.

The problem with the first screen line of a window is the call to
move_it_by_lines with its second argument zero.  The simplest solution
is to eliminate that call, and instead achieve the same effect "by
hand" (if at all necessary).

> To move to the right column of the line necessarily involves knowing
> where the "actual" BOL is.

No, not necessarily.  Moreover, the "actual BOL" will not help you if
it is known as a buffer position (as opposed to its screen horizontal
coordinate).

> At the moment, I can't see how post analysis (i.e. analysis after
> Fvertical_motion has done move_it_by_lines (&it, nlines)) can be
> avoided.

I don't understand the significance of what you want to say here,
sorry.  One of the large changes I made does precisely that, right?

> In this line of thinking, considering how bidi complicates the order of
> PT, end position, and various BOLs, it might be very convenient to
> enhance move_it_in_display_line_to to take TWO (or even several)
> to_charpos parameters, such that the function would stop after reaching
> either one of them.

IME, this kind of reasoning about these problems is not useful, it
leads into a conceptual trap.  The key problem is with the definition
of "reaching".  In the unidirectional display, whenever you see a
buffer position >= goal, you have "reached" the goal.  But this breaks
with bidirectional text, so only strict equality can be meaningfully
tested.  But the exact equality might legitimately be out of our reach
(pun intended), for any number of valid reasons: display properties,
truncated lines, position outside the visible portion of the buffer,
and many others.  So you can never "reach" the goal, but it doesn't
mean you didn't find your spot.  And there are no good solutions to
this conundrum.

Instead of reasoning in terms of buffer positions, you should reason
in terms of screen coordinates.  The move_it_* functions all work in
the visual order, i.e. they traverse the text in the order it would be
displayed, from left to right, screen line after screen line.  If you
think in terms of screen coordinates and horizontal/vertical moves
across screen lines, you can blissfully forget about bidi.

> > I hope the changes I made and the comments to those changes will speak
> > for themselves, and show how to solve such problems.
> 
> I like your idea for solving the "going back over the top of the window"
> problem.  Is there any guarantee that w->prev will actually be the
> window "to the left of" w (whatever that might mean)?

Yes.  See the documentation and the implementation of
window-prev-sibling and window-next-sibling.

> > But in general, I must say that fixing what you are trying to fix
> > is an ambitious undertaking; perhaps it would be better to make
> > sure the two windows are always the same width, using pixel-wise
> > resizing where necessary.
> 
> As already said, this would be sub-optimal for ttys, where the total
> width is fixed, and tends not to be a number of the form 2n+1 or 3n+2.

We could say that on a TTY vertical cursor motion across windows can
create strange artifacts, if a continued line spans the window.  It's
a relatively rare situation, I think.

Or maybe we could arrange for the windows to be scrolled when cursor
approaches the edge, so that a continued line never straddles more
than one window.

Any of these 2 possibilities looks better to me than to fight an
uphill battle against hidden assumptions within the display engine.
The more I think about this, the more it feels like a tip of a giant
iceberg.  Beware!

Re: "... the window start at a meaningless point within a line."

[Eli Zaretskii]

Btw, you said you ignore display and overlay strings for now, but
their existence raises a serious problem wrt your "exact window-start"
solution: what if a display string starts on one screen line and ends
on the next one, i.e. the continuation happens in the middle of a
display string?  In this case, you will not be able to force a
window-start on a buffer position that is "covered" by the display
string, so some part of the string will either be displayed twice, or
not at all.

This is also a problem with today's unmodified Follow mode: when such
a string is between two windows, the second portion of the string (the
one in the second window) is not displayed at all.

Re: "... the window start at a meaningless point within a line."

[Alan Mackenzie]

Hello, Eli.

On Sun, Nov 01, 2015 at 07:45:28PM +0200, Eli Zaretskii wrote:
> > Date: Sun, 1 Nov 2015 15:23:54 +0000
> > Cc: emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>
> > 
> > > The patch below should go a long way towards solving these problems.
> > > The only issue I left unsolved is exposed when typing C-p on the
> > > second screen line of the right-side window: you will see that it
> > > doesn't succeed in keeping the visual column.  Solving this is left as
> > > an exercise ;-)
> > 
> > ;-).  Sometimes it doesn't even go to the right line (i.e. when there
> > are more or fewer "actual" lines than "xdisp" lines).

> The code is explicitly unfinished, I said that much.  I wrote it in
> the hope that it will show enough to get you off the ground with using
> the move_it_* functions for the purposes such as this.  But it
> definitely needs more work.

> > To detect this situation necessarily involves reseat_at_window_start
> > followed by moving forward, counting (display) line starts, in some
> > fashion.  Do you agree?

> I don't know if I agree.  I don't think I understand the situation
> well enough to reason about it.

> The problem with the first screen line of a window is the call to
> move_it_by_lines with its second argument zero.  The simplest solution
> is to eliminate that call, and instead achieve the same effect "by
> hand" (if at all necessary).

> > To move to the right column of the line necessarily involves knowing
> > where the "actual" BOL is.

> No, not necessarily.  Moreover, the "actual BOL" will not help you if
> it is known as a buffer position (as opposed to its screen horizontal
> coordinate).

> > At the moment, I can't see how post analysis (i.e. analysis after
> > Fvertical_motion has done move_it_by_lines (&it, nlines)) can be
> > avoided.

> I don't understand the significance of what you want to say here,
> sorry.  One of the large changes I made does precisely that, right?

> > In this line of thinking, considering how bidi complicates the order of
> > PT, end position, and various BOLs, it might be very convenient to
> > enhance move_it_in_display_line_to to take TWO (or even several)
> > to_charpos parameters, such that the function would stop after reaching
> > either one of them.

> IME, this kind of reasoning about these problems is not useful, it
> leads into a conceptual trap.  The key problem is with the definition
> of "reaching".  In the unidirectional display, whenever you see a
> buffer position >= goal, you have "reached" the goal.  But this breaks
> with bidirectional text, so only strict equality can be meaningfully
> tested.  But the exact equality might legitimately be out of our reach
> (pun intended), for any number of valid reasons: display properties,
> truncated lines, position outside the visible portion of the buffer,
> and many others.  So you can never "reach" the goal, but it doesn't
> mean you didn't find your spot.  And there are no good solutions to
> this conundrum.

> Instead of reasoning in terms of buffer positions, you should reason
> in terms of screen coordinates.  The move_it_* functions all work in
> the visual order, i.e. they traverse the text in the order it would be
> displayed, from left to right, screen line after screen line.  If you
> think in terms of screen coordinates and horizontal/vertical moves
> across screen lines, you can blissfully forget about bidi.

I'm beginning to come around to what you suggested some while ago,
namely making Follow Mode windows as wide as eachother.  While I'm
confident I could make the maybe_move_to_exact_bol approach, modified to
use screen positions instead of buffer positions, work properly, it
would take a lot of work.  Probably more work than it's worth.

So I'm going to be breaking off this line of work, at least for now, and
look at equal width windows.

> > > I hope the changes I made and the comments to those changes will speak
> > > for themselves, and show how to solve such problems.
> > 
> > I like your idea for solving the "going back over the top of the window"
> > problem.  Is there any guarantee that w->prev will actually be the
> > window "to the left of" w (whatever that might mean)?

> Yes.  See the documentation and the implementation of
> window-prev-sibling and window-next-sibling.

OK, thanks.

> > > But in general, I must say that fixing what you are trying to fix
> > > is an ambitious undertaking; perhaps it would be better to make
> > > sure the two windows are always the same width, using pixel-wise
> > > resizing where necessary.
> > 
> > As already said, this would be sub-optimal for ttys, where the total
> > width is fixed, and tends not to be a number of the form 2n+1 or 3n+2.

> We could say that on a TTY vertical cursor motion across windows can
> create strange artifacts, if a continued line spans the window.  It's
> a relatively rare situation, I think.

I think it's more rare that anybody notices, but it happens quite a lot.

> Or maybe we could arrange for the windows to be scrolled when cursor
> approaches the edge, so that a continued line never straddles more
> than one window.

> Any of these 2 possibilities looks better to me than to fight an
> uphill battle against hidden assumptions within the display engine.
> The more I think about this, the more it feels like a tip of a giant
> iceberg.  Beware!

OK.  I give up on this for now.  I'll look at width equality.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: "... the window start at a meaningless point within a line."

[Alan Mackenzie]

Hello, Eli.

On Sat, Oct 17, 2015 at 04:31:05PM +0300, Eli Zaretskii wrote:
> > Date: Sat, 17 Oct 2015 15:34:44 +0300
> > From: Eli Zaretskii <eliz@gnu.org>
> > Cc: emacs-devel@gnu.org

> > So now I think that the problem can be solved as follows:

> >   . if Fvertical_motion's code without changes ends up before the
> >     first newline that follows the window-start, you don't need to do
> >     anything, because Fvertical_motion does exactly what you want in
> >     that case

> >   . otherwise Fvertical_motion will either end up on the screen line
> >     before the goal, or will be on the right screen line, but to the
> >     right of the goal column, and a correction should be applied by
> >     moving point to the left

> Is the following a correct definition of when Fvertical_motion
> overshoots, i.e. lands one line below (for LINES > 0) the correct one:
> when the offset between the "actual" window-start and the preceding
> "xdisp" BOL is greater than the X coordinate of the last character of
> the 1st physical line in the window.

Disentangling the words, what I think you're saying is that ....

   nlines = 3
8. B-------WS-------------A-------L1-------------A2----\nC--------C2--------C3
   <---N--->                      <---------X--------->
(a)                           ^                                   it,T
(b)                                  ^                            it        T


   nlines = 3
9. B-----------------WS---A-----------------L1---A2----\nC--------C2--------C3
   <--------N-------->                      <----X---->
(a)                           ^                                   it,T
(b)                                            ^                  it        T

.... an overshoot occurs when N > X.  I'm assuming that by "X coordinate
of ..." you're meaning the offset from an _actual_ BOL.

I don't see at all that this is the case.  In diagram 8, N < X, and in
diagram 9, N > X.  In 8(a) and 9(a) Fvertical_motion hits the target
directly.  In 8(b) and 9(b), there is an "undershoot" rather than an
"overshoot".

The critical factor is not whether N > X; it's the precise positioning
of point with respect to the actual and xdisp BOLs on the first text
line.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: "... the window start at a meaningless point within a line."

[Eli Zaretskii]

> Date: Sun, 18 Oct 2015 14:53:56 +0000
> Cc: emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > Is the following a correct definition of when Fvertical_motion
> > overshoots, i.e. lands one line below (for LINES > 0) the correct one:
> > when the offset between the "actual" window-start and the preceding
> > "xdisp" BOL is greater than the X coordinate of the last character of
> > the 1st physical line in the window.
> 
> Disentangling the words, what I think you're saying is that ....
> 
>    nlines = 3
> 8. B-------WS-------------A-------L1-------------A2----\nC--------C2--------C3
>    <---N--->                      <---------X--------->
> (a)                           ^                                   it,T
> (b)                                  ^                            it        T
> 
> 
>    nlines = 3
> 9. B-----------------WS---A-----------------L1---A2----\nC--------C2--------C3
>    <--------N-------->                      <----X---->
> (a)                           ^                                   it,T
> (b)                                            ^                  it        T
> 
> .... an overshoot occurs when N > X.  I'm assuming that by "X coordinate
> of ..." you're meaning the offset from an _actual_ BOL.
> 
> I don't see at all that this is the case.

We are miscommunicating (because these diagrams are devilishly hard to
interpret).

But let's leave this alone, and instead try to use my other idea.

Re: "... the window start at a meaningless point within a line."

[Alan Mackenzie]

Hello, Eli.

On Sun, Oct 18, 2015 at 08:46:48PM +0300, Eli Zaretskii wrote:
> > Date: Sun, 18 Oct 2015 14:53:56 +0000
> > Cc: emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

[ .... ]

> >    nlines = 3
> > 8. B-------WS-------------A-------L1-------------A2----\nC--------C2--------C3
> >    <---N--->                      <---------X--------->
> > (a)                           ^                                   it,T
> > (b)                                  ^                            it        T

[ .... ]

> We are miscommunicating (because these diagrams are devilishly hard to
> interpret).

Sorry, I didn't realise.  Obviously they're easy for me.  Could we
possibly improve them somehow, since I find it difficult to think things
through without some sort of diagram?

For example, would it help if the actual BOLs and the xdisp BOLs were
written on separate lines?  Something like this:

      nlines = 3
Act:         WS                    L1                    C      C2     C3
8.    --------------------------^----------------------\n-------it,T-------
Xdi:  B                     A                    A2      C      C2     C3

, where ^ is the starting point, T is the target BOL and "it" is where
Fvertical_motion moves `it' to?

[ .... ]

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: "... the window start at a meaningless point within a line."

[Eli Zaretskii]

> Date: Mon, 19 Oct 2015 10:45:07 +0000
> Cc: emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > We are miscommunicating (because these diagrams are devilishly hard to
> > interpret).
> 
> Sorry, I didn't realise.  Obviously they're easy for me.  Could we
> possibly improve them somehow, since I find it difficult to think things
> through without some sort of diagram?
> 
> For example, would it help if the actual BOLs and the xdisp BOLs were
> written on separate lines?

Not really, not for me, anyway.

My problem is that you "flatten" the layout: where the actual layout
is 2-dimensional, you make it 1-dimensional, and replace visual cues
with those A2, L3 markers, that I can never remember which one is
which.  Reasoning about VERTICAL motion while using a HORIZONTAL
diagram doesn't help, either.

So if you want to use diagrams, let's make them reflect the screen
layout, like this:

  B--------------------WS----------
  A--------------------L1----------
  A2------\n
  C---------------------------\n
  C2-------------------------\n

etc., where a line without a \n at the end is continued on the next
one.

With this arrangement, the symbol's at the left can be discarded, and
we only need the ones that are in the middle of the lines.

If it's easier for you to have the lines start at WS and L1 instead of
B and A, that's okay, too; we should just agree which convention we
use.

Re: "... the window start at a meaningless point within a line."

[Alan Mackenzie]

Hello, Eli.

On Mon, Oct 19, 2015 at 01:56:22PM +0300, Eli Zaretskii wrote:
> > Date: Mon, 19 Oct 2015 10:45:07 +0000
> > Cc: emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

[ .... ]

> My problem is that you "flatten" the layout: where the actual layout
> is 2-dimensional, you make it 1-dimensional, and replace visual cues
> with those A2, L3 markers, that I can never remember which one is
> which.  Reasoning about VERTICAL motion while using a HORIZONTAL
> diagram doesn't help, either.

> So if you want to use diagrams, let's make them reflect the screen
> layout, like this:

>   B--------------------WS----------
>   A--------------------L1----------
>   A2------\n
>   C---------------------------\n
>   C2-------------------------\n

> etc., where a line without a \n at the end is continued on the next
> one.

> With this arrangement, the symbol's at the left can be discarded, and
> we only need the ones that are in the middle of the lines.

> If it's easier for you to have the lines start at WS and L1 instead of
> B and A, that's okay, too; we should just agree which convention we
> use.

I think I'd prefer to have the "actual" BOLs at the LHS.  But the
symbols at the left are still useful for discussion about a scenario.
For point, `it', and target, I suggest we write these below the line
they apply to, even though they look a little ambiguous there.  So, how
about this:

8. nlines = 3:
      B-------------------------------
WS----A-------------------------------
L1----A2------------------------------\n
   ^
C-------------------------------------
C2------------------------------------\n
it
C3------------------------------------
T

?

Here, point starts between L1 and A2, our target BOL is C3 (3 actual
BOLs after L1) and Fvertical_motion puts `it' at C2 (3 xdisp BOLs after
A).

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: "... the window start at a meaningless point within a line."

[Eli Zaretskii]

> Date: Mon, 19 Oct 2015 11:24:25 +0000
> Cc: emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> I think I'd prefer to have the "actual" BOLs at the LHS.  But the
> symbols at the left are still useful for discussion about a scenario.

If you insist.  But then please make them consistent: there's no need
to start with B, then go to A, then to C, then to C2 (what happened to
C1?), etc.

> 8. nlines = 3:
>       B-------------------------------
> WS----A-------------------------------
> L1----A2------------------------------\n
>    ^
> C-------------------------------------
> C2------------------------------------\n
> it
> C3------------------------------------
> T
> 
> ?
> 
> Here, point starts between L1 and A2, our target BOL is C3 (3 actual
> BOLs after L1) and Fvertical_motion puts `it' at C2 (3 xdisp BOLs after
> A).

I don't understand why "T" is always at bol.  It can be any column.

Re: "... the window start at a meaningless point within a line."

[Alan Mackenzie]

Hello, Eli.

On Mon, Oct 19, 2015 at 02:28:54PM +0300, Eli Zaretskii wrote:
> > Date: Mon, 19 Oct 2015 11:24:25 +0000
> > Cc: emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>
> > 
> > I think I'd prefer to have the "actual" BOLs at the LHS.  But the
> > symbols at the left are still useful for discussion about a scenario.

> If you insist.  But then please make them consistent: there's no need
> to start with B, then go to A, then to C, then to C2 (what happened to
> C1?), etc.

:-)  They're not that consistent, are they?  "B" and "A" were the xdisp
BOLs "B"efore and "A"fter WS.  "C" was for "C"ommon lines.  I must have
been starting counting at 1 when the next symbol was "C2".

But I think, for discussion, it's useful to maintain the symbolic
distinction between actual, xdisp, and common BOLs.  How about using WS,
A2, A3, ... for actual BOLs, X0, X1, X2, ... for xdisp bols, and C1,
C2, ... for the common ones?  So we'd end up with this:

8. nlines = 3:
      X0------------------------------
WS----X1------------------------------
A2----X2------------------------------\n
   ^
C1------------------------------------
C2------------------------------------\n
it
C3------------------------------------
T

That leaves the possibility of X-1, X-2 when we need to go backwards
from WS, and also the possibility of A0, A-1 for actual BOLs before WS,
though I'm not sure what that might mean.

> > Here, point starts between L1 and A2, our target BOL is C3 (3 actual
> > BOLs after L1) and Fvertical_motion puts `it' at C2 (3 xdisp BOLs after
> > A).

> I don't understand why "T" is always at bol.  It can be any column.

I'm only really looking at the problem of getting to the correct
beginning of line.  Moving to a specific column (with a cons argument to
vertical-motion) seems to be less of a problem.  That seems to work OK.
(By the way, have you tried the code at all?)

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: "... the window start at a meaningless point within a line."

[Eli Zaretskii]

> Date: Mon, 19 Oct 2015 12:02:23 +0000
> Cc: emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > I don't understand why "T" is always at bol.  It can be any column.
> 
> I'm only really looking at the problem of getting to the correct
> beginning of line.  Moving to a specific column (with a cons argument to
> vertical-motion) seems to be less of a problem.

If that's in a continuation line, you have the same problem: the
display engine and the move_it_* functions count coordinates and
columns from their BOL, not from your "actual" BOLs.

> (By the way, have you tried the code at all?)

No, because I don't know how much I need to just see the problems,
without the solutions.  At some point I will probably ask you for a
minimal patch an a recipe to see the problem, unless we will find a
solution by just talking.

Re: "... the window start at a meaningless point within a line."

[Alan Mackenzie]

Hello, Eli.

On Mon, Oct 19, 2015 at 03:33:40PM +0300, Eli Zaretskii wrote:
> > Date: Mon, 19 Oct 2015 12:02:23 +0000
> > Cc: emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > > I don't understand why "T" is always at bol.  It can be any column.

> > I'm only really looking at the problem of getting to the correct
> > beginning of line.  Moving to a specific column (with a cons argument to
> > vertical-motion) seems to be less of a problem.

> If that's in a continuation line, you have the same problem: the
> display engine and the move_it_* functions count coordinates and
> columns from their BOL, not from your "actual" BOLs.

That's not what I see.  Having set `it' in Fvertical_motion to an
iterator based at WS, the column movement remains based on `it', i.e.
based on "actual" BOLs.

> > (By the way, have you tried the code at all?)

> No, because I don't know how much I need to just see the problems,
> without the solutions.  At some point I will probably ask you for a
> minimal patch an a recipe to see the problem, unless we will find a
> solution by just talking.

OK.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: "... the window start at a meaningless point within a line."

[Eli Zaretskii]

> Date: Mon, 19 Oct 2015 13:11:58 +0000
> Cc: emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> > > I'm only really looking at the problem of getting to the correct
> > > beginning of line.  Moving to a specific column (with a cons argument to
> > > vertical-motion) seems to be less of a problem.
> 
> > If that's in a continuation line, you have the same problem: the
> > display engine and the move_it_* functions count coordinates and
> > columns from their BOL, not from your "actual" BOLs.
> 
> That's not what I see.  Having set `it' in Fvertical_motion to an
> iterator based at WS, the column movement remains based on `it', i.e.
> based on "actual" BOLs.

When move_it_by_lines returns, you are on the goal line, at zero X
coordinate.  That zero is in "xdisp coordinates".

Re: "... the window start at a meaningless point within a line."

[Alan Mackenzie]

Hello, Eli.

On Mon, Oct 19, 2015 at 04:27:38PM +0300, Eli Zaretskii wrote:
> > Date: Mon, 19 Oct 2015 13:11:58 +0000
> > Cc: emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > > > I'm only really looking at the problem of getting to the correct
> > > > beginning of line.  Moving to a specific column (with a cons argument to
> > > > vertical-motion) seems to be less of a problem.

> > > If that's in a continuation line, you have the same problem: the
> > > display engine and the move_it_* functions count coordinates and
> > > columns from their BOL, not from your "actual" BOLs.

> > That's not what I see.  Having set `it' in Fvertical_motion to an
> > iterator based at WS, the column movement remains based on `it', i.e.
> > based on "actual" BOLs.

> When move_it_by_lines returns, you are on the goal line, at zero X
> coordinate.  That zero is in "xdisp coordinates".

OK, I see it.  There's a "move_by_lines (it, -1)" near the end of
maybe_move_to_exact_bol.  Usually, this would be OK, since we'd be
dealing with "common" BOLs rather than "actual" BOLs, but there might be
a borderline case where `it' would need to go up to the last "actual"
BOL.  I'll need to look at this.

Thanks.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: "... the window start at a meaningless point within a line."

[Alan Mackenzie]

Hello, Eli.

On Mon, Oct 19, 2015 at 07:15:22PM +0000, Alan Mackenzie wrote:
> On Mon, Oct 19, 2015 at 04:27:38PM +0300, Eli Zaretskii wrote:
> > > Date: Mon, 19 Oct 2015 13:11:58 +0000
> > > Cc: emacs-devel@gnu.org
> > > From: Alan Mackenzie <acm@muc.de>

> > > > > I'm only really looking at the problem of getting to the correct
> > > > > beginning of line.  Moving to a specific column (with a cons argument to
> > > > > vertical-motion) seems to be less of a problem.

> > > > If that's in a continuation line, you have the same problem: the
> > > > display engine and the move_it_* functions count coordinates and
> > > > columns from their BOL, not from your "actual" BOLs.

> > > That's not what I see.  Having set `it' in Fvertical_motion to an
> > > iterator based at WS, the column movement remains based on `it', i.e.
> > > based on "actual" BOLs.

> > When move_it_by_lines returns, you are on the goal line, at zero X
> > coordinate.  That zero is in "xdisp coordinates".

> OK, I see it.  There's a "move_by_lines (it, -1)" near the end of
> maybe_move_to_exact_bol.  Usually, this would be OK, since we'd be
> dealing with "common" BOLs rather than "actual" BOLs, but there might be
> a borderline case where `it' would need to go up to the last "actual"
> BOL.  I'll need to look at this.

This "move_by_lines (it, -1)" isn't a problem, since the condition for
it occurring ensures it is only executed when starting from a "common"
BOL later than the first common BOL.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: "... the window start at a meaningless point within a line."

[Alan Mackenzie]

Hello, Eli.

On Sat, Oct 17, 2015 at 03:34:44PM +0300, Eli Zaretskii wrote:
> > Date: Sat, 17 Oct 2015 11:57:38 +0000
> > Cc: emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > > > >   . compute the horizontal difference, in pixels, between the position
> > > > >     which xdisp would use as window-start and the actual window-start
> > > > >     (the value should always be positive); let's call the result N

> > Just a point I missed last night: N could be either positive or negative,
> > depending on whether the "exact" window start is just before or just
> > after the "xdisp" window start.  A situation where N would be negative
> > is:

> > 3.  B---------------WS---A--------------L1---A2---------\nC---------C2---

> > , where the "xdisp" window start would be at A, not B.

> No, it is always positive, because in my definition the xdisp start is
> at B.

You wrote ".... the position which xdisp would use as window-start ...".
This position is either A or B, whichever is closer to the requested WS
in `set-window-start'.  The code which calculates it is (before my patch)
in compute_window_start_on_continuation_line, called directly from
redisplay_window.

In my patch, I extracted get_window_start_on_continuation_line from the
above function, and call it from Fvertical_motion to determine this
position.

We could redefine the "xdisp window start" as always being at B, but that
would involve sometimes having to go a line backward from the position
returned by get_window_start_on_continuation_line.  I'm not convinced
that this redefinition would be more helpful than confusing.  I think if
we were to do this, then we should use a more appropriate term than
"xdisp window start". 

> It cannot start at A, because that would put point outside of the
> window.

The redisplay code is actually quite happy about point being at WS when A
is the "xdisp window start" (I've tried it out, many times).  I think
this code uses w->start to test whether point is "outside the window".

> IOW, we always compute the offset relative to the BOL of a screen line
> that includes the "actual" window start.

OK, I'll take that as a given for the rest of this email.

> > (BTW, I'm not all that enthusiastic about my terminology "exact" and
> > "xdisp" here, and if anybody could come up with better suggestions, I'd
> > welcome it.  But "exact" and "xdisp" are, at the very least, quite
> > short.)

> "Exact" is a problematic word; I've been using "actual" instead.

Yes, I think that's a little bit better.  If it ever comes to the point
where my existing code (however modified) looks like getting committed,
I'll change all the comments and variable names to "*actual*".

> > > Starting from a BOL is a special case that can be handled by special
> > > code.  Will the algorithm I suggested work in all other cases?

> > Starting at a(n xdisp) BOL isn't a special case: anywhere between A
> > (inclusive) and L1 (exclusive) is equivalent.

> After the correction of the definition of the "xdisp" BOL above, do
> you still disagree that only starting from a BOL is special?
> (Actually, I now think that even that situation isn't special, see
> below.)

I was never actually convinced that even in the interval [A L1) was
special in any way.

> > > My analysis of what you described concluded, perhaps wrongly, that the
> > > essence of the problem is the constant offset between the "actual"
> > > beginning of each screen line and the display engine's idea of that
> > > beginning.  This constant offset is limited to the first physical line
> > > of the window; once we get past the first newline, the offset
> > > disappears.  Is this conclusion correct?

> > I don't think so.  That constant offset, N, is indeed relevant in the
> > first text line in the window.  But after the first newline, whether or
> > not we have to move one up or down from Fvertical_motion's position is
> > critically dependent upon the arrangement of exact and xdisp BOLs in that
> > first text line.

> Once again, do you still think this after correcting the definition of
> the "xdisp" window start?

I'll have to study that more carefully and get back to you.  But
intuitively, I still think that what I said holds.

> Btw, I was wrong in that the correction involves movement to the
> right.  It actually calls for a movement to the left, which is
> slightly more complicated (move_it_* functions cannot do that, so you
> need to go to the beginning of a screen line first), but not too
> complex.

I'll have to draw another picture, here.  (I'll get you drawing these
pictures too, before the end of this conversation.  :-)

    nlines = -1
4.  B---WS------------A---L1------------A2------------L2-----\nC-----
                      it  T                           ^

Here, if point starts at L2, our target is clearly L1, one line back from
the "actual" BOL we start from.  Fvertical_motion first moves back to the
previous "xdisp" BOL, A2, then to the "xdisp" BOL before that, A1.  Our
correction would be N pixels to the _right_, as you originally stated.

On the other hand, if our starting point is one character back,

    nlines = -1
5.  B---WS------------A---L1------------A2------------L2-----\nC-----
        T             it                             ^ 

, then our target is now WS, but Fvertical_motion behaves exactly the
same, putting `it' at A.  So now our correction is (window_width - N)
characters backwards.  Or, put another way we need to go a line backwards
(to B) and then N pixels forwards.

I thus don't agree that the N pixel correction involves movement to the
right.

So, even when our target "actual" BOL is on the first text line on the
window, we need to pay attention to our exact starting position - in
particular, whether or not there is an "actual" BOL between the previous
"xdisp" BOL and point.

> So now I think that the problem can be solved as follows:

>   . if Fvertical_motion's code without changes ends up before the
>     first newline that follows the window-start, you don't need to do
>     anything, because Fvertical_motion does exactly what you want in
>     that case

Er, hang on.  By "before" I understand "at a smaller buffer position
than".  This is surely the case where we need the N pixel adjustment,
either on the line Fv_m puts us on, or the previous one.

>   . otherwise Fvertical_motion will either end up on the screen line
>     before the goal, or will be on the right screen line, but to the
>     right of the goal column, and a correction should be applied by
>     moving point to the left

> Do you agree?

Emphatically no.  It is essential to analyse the starting position too,
as illustrated by the difference between examples 4 and 5.

I think you got the binary criterion there reversed, and really meant "if
Fv_m's code without changes ends up AFTER the first newline ....".  But
in this case, sometimes Fv_m takes us to the correct target line,
sometimes it's one line out.  (See example 1 in my email from Fri, 16 Oct
2015 20:12:38 +0000).

-- 
Alan Mackenzie (Nuremberg, Germany).