bug#23098: 25.0.92; (recenter -1) can leave point in the middle of the window

bug#23098: 25.0.92; (recenter -1) can leave point in the middle of the window

[Jorgen Schaefer]

Hello!

Evaluating the following code in "emacs -Q" leaves point in the middle
of the window, instead of at the end of the window as expected. Certain
modes (IRC) want to keep the buffer flushed at the end of the window,
this problem interferes with that behavior and causes very weird jumpy
scrolling.


(require 'cl)
(let* ((b (get-buffer-create "*Bug Repro*"))
       (w (selected-window))
       (l (window-height w))
       o)
  (set-window-buffer w b)
  (select-window w)
  (set-buffer b)
  (erase-buffer)
  (cl-dotimes (i (* l 2))
              (insert "foo\n"))
  (goto-char (point-min))
  (forward-line (+ l 2))
  (setq o (make-overlay (point) (point)))
  (overlay-put o 'after-string "\n")
  (goto-char (point-max))
  (recenter -1))


That is, if there is a multi-line overlay that would be shown only
halfway.

I have seen similar behavior with some forms of invisible text, but have
not been able to create reproduction steps for that.

Regards,
Jorgen


In GNU Emacs 25.0.92.1 (x86_64-unknown-linux-gnu)
 of 2016-03-14 built on loki.jorgenschaefer.de
Repository revision: 1b9d6163b023aaefd15d38ea28e968a113202402
System Description:	Debian GNU/Linux 8.3 (jessie)

Configured using:
 'configure --enable-checking --without-x 'CFLAGS=-ggdb3 -O0'
 LDFLAGS=-ggdb3'

bug#23098: 25.0.92; (recenter -1) can leave point in the middle of the window

[Eli Zaretskii]

> From: Jorgen Schaefer <jorgen.schaefer@gmail.com>
> Date: Wed, 23 Mar 2016 19:40:27 +0100
> 
> Evaluating the following code in "emacs -Q" leaves point in the middle
> of the window, instead of at the end of the window as expected. Certain
> modes (IRC) want to keep the buffer flushed at the end of the window,
> this problem interferes with that behavior and causes very weird jumpy
> scrolling.
> 
> 
> (require 'cl)
> (let* ((b (get-buffer-create "*Bug Repro*"))
>        (w (selected-window))
>        (l (window-height w))
>        o)
>   (set-window-buffer w b)
>   (select-window w)
>   (set-buffer b)
>   (erase-buffer)
>   (cl-dotimes (i (* l 2))
>               (insert "foo\n"))
>   (goto-char (point-min))
>   (forward-line (+ l 2))
>   (setq o (make-overlay (point) (point)))
>   (overlay-put o 'after-string "\n")
>   (goto-char (point-max))
>   (recenter -1))
> 
> 
> That is, if there is a multi-line overlay that would be shown only
> halfway.

I'm not sure I understand: are you trying to display a window whose
start point is in the middle of an overlay string?  That's not
possible, AFAIK.  The window-start point must be a buffer position.

bug#23098: 25.0.92; (recenter -1) can leave point in the middle of the window

[Jorgen Schäfer]

[-- Attachment #1: Type: text/plain, Size: 1947 bytes --]

Hello!
The use case here are buffers like comint or IRC chat buffers. It's nice to
keep the last line of the buffer flush with the last line of the window, to
show the maximum amount of data. (recenter -1) at the end of a buffer is
supposed to do just that. In this case, it does not, but instead leaves
point in the *middle* of the window.

If it is not possible to display that line halfway (which sounds like a
missing feature to begin with), I would expect Emacs to least try to leave
point as low in the window as possible, instead of in the middle of the
window. C-l likewise won't move point any further down, for example.

Regards,
Jorgen

On Wed, Mar 23, 2016 at 8:18 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Jorgen Schaefer <jorgen.schaefer@gmail.com>
> > Date: Wed, 23 Mar 2016 19:40:27 +0100
> >
> > Evaluating the following code in "emacs -Q" leaves point in the middle
> > of the window, instead of at the end of the window as expected. Certain
> > modes (IRC) want to keep the buffer flushed at the end of the window,
> > this problem interferes with that behavior and causes very weird jumpy
> > scrolling.
> >
> >
> > (require 'cl)
> > (let* ((b (get-buffer-create "*Bug Repro*"))
> >        (w (selected-window))
> >        (l (window-height w))
> >        o)
> >   (set-window-buffer w b)
> >   (select-window w)
> >   (set-buffer b)
> >   (erase-buffer)
> >   (cl-dotimes (i (* l 2))
> >               (insert "foo\n"))
> >   (goto-char (point-min))
> >   (forward-line (+ l 2))
> >   (setq o (make-overlay (point) (point)))
> >   (overlay-put o 'after-string "\n")
> >   (goto-char (point-max))
> >   (recenter -1))
> >
> >
> > That is, if there is a multi-line overlay that would be shown only
> > halfway.
>
> I'm not sure I understand: are you trying to display a window whose
> start point is in the middle of an overlay string?  That's not
> possible, AFAIK.  The window-start point must be a buffer position.
>

[-- Attachment #2: Type: text/html, Size: 2623 bytes --]

bug#23098: 25.0.92; (recenter -1) can leave point in the middle of the window

[Eli Zaretskii]

> From: Jorgen Schäfer <jorgen.schaefer@gmail.com>
> Date: Wed, 23 Mar 2016 19:40:50 +0000
> Cc: 23098@debbugs.gnu.org
> 
> The use case here are buffers like comint or IRC chat buffers. It's nice to keep the last line of the buffer flush
> with the last line of the window, to show the maximum amount of data. (recenter -1) at the end of a buffer is
> supposed to do just that. In this case, it does not, but instead leaves point in the *middle* of the window.

Yes, because the scrolling fails, and Emacs falls back on recentering.

> If it is not possible to display that line halfway (which sounds like a missing feature to begin with), I would
> expect Emacs to least try to leave point as low in the window as possible, instead of in the middle of the
> window. C-l likewise won't move point any further down, for example.

If you set scroll-conservatively to a value larger than 100, don't you
get point as low as possible?

bug#23098: 25.0.92; (recenter -1) can leave point in the middle of the window

[Jorgen Schäfer]

[-- Attachment #1: Type: text/plain, Size: 1181 bytes --]

> If you set scroll-conservatively to a value larger than 100, don't you
> get point as low as possible?

No.

On Wed, Mar 23, 2016 at 9:22 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Jorgen Schäfer <jorgen.schaefer@gmail.com>
> > Date: Wed, 23 Mar 2016 19:40:50 +0000
> > Cc: 23098@debbugs.gnu.org
> >
> > The use case here are buffers like comint or IRC chat buffers. It's nice
> to keep the last line of the buffer flush
> > with the last line of the window, to show the maximum amount of data.
> (recenter -1) at the end of a buffer is
> > supposed to do just that. In this case, it does not, but instead leaves
> point in the *middle* of the window.
>
> Yes, because the scrolling fails, and Emacs falls back on recentering.
>
> > If it is not possible to display that line halfway (which sounds like a
> missing feature to begin with), I would
> > expect Emacs to least try to leave point as low in the window as
> possible, instead of in the middle of the
> > window. C-l likewise won't move point any further down, for example.
>
> If you set scroll-conservatively to a value larger than 100, don't you
> get point as low as possible?
>

[-- Attachment #2: Type: text/html, Size: 1633 bytes --]

bug#23098: 25.0.92; (recenter -1) can leave point in the middle of the window

[Eli Zaretskii]

> From: Jorgen Schäfer <jorgen.schaefer@gmail.com>
> Date: Thu, 24 Mar 2016 08:12:08 +0000
> Cc: 23098@debbugs.gnu.org
> 
> > If you set scroll-conservatively to a value larger than 100, don't you
> > get point as low as possible?
> 
> No.

That's strange.

Let me be absolutely sure we are trying the same thing.  What I did is
the following:

  . emacs -Q
  . paste the following into *scratch*, which is exactly the code you
    posted with a single line added:

    (require 'cl)
    (let* ((b (get-buffer-create "*Bug Repro*"))
	   (w (selected-window))
	   (l (window-height w))
	   o)
      (set-window-buffer w b)
      (select-window w)
      (set-buffer b)
      (erase-buffer)
      (setq scroll-conservatively 101) ;; <<<<< this line was added
      (cl-dotimes (i (* l 2))
		  (insert "foo\n"))
      (goto-char (point-min))
      (forward-line (+ l 2))
      (setq o (make-overlay (point) (point)))
      (overlay-put o 'after-string "\n")
      (goto-char (point-max))
      (recenter -1))

  . M-x eval-region RET

After this, buffer "*Bug Repro*" pops up, with point at EOB positioned
on the 2nd line from bottom of the window (instead of the last line).

If you do the above, do you still get the cursor recentered in the
window?  If so, I will try to play with display defaults in order to
reproduce what you see.

Thanks.

bug#23098: 25.0.92; (recenter -1) can leave point in the middle of the window

[Jorgen Schäfer]

[-- Attachment #1: Type: text/plain, Size: 2171 bytes --]

> If you do the above, do you still get the cursor recentered in the
> window?

Hm. That seems to work. I was using let-binding around recenter, which does
not work. Even resetting scroll-conservatively after the call to recenter
will exhibit the bug again. So setting scroll-conservatively is a possible
workaround, although it means that *all* scrolling will be affected by
scroll-conservatively. E.g. scrolling through the backlog of a M-x shell
session would be affected, even though it just should affect the scrolling
when the last line of the buffer is visible at all.

On Thu, Mar 24, 2016 at 4:48 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Jorgen Schäfer <jorgen.schaefer@gmail.com>
> > Date: Thu, 24 Mar 2016 08:12:08 +0000
> > Cc: 23098@debbugs.gnu.org
> >
> > > If you set scroll-conservatively to a value larger than 100, don't you
> > > get point as low as possible?
> >
> > No.
>
> That's strange.
>
> Let me be absolutely sure we are trying the same thing.  What I did is
> the following:
>
>   . emacs -Q
>   . paste the following into *scratch*, which is exactly the code you
>     posted with a single line added:
>
>     (require 'cl)
>     (let* ((b (get-buffer-create "*Bug Repro*"))
>            (w (selected-window))
>            (l (window-height w))
>            o)
>       (set-window-buffer w b)
>       (select-window w)
>       (set-buffer b)
>       (erase-buffer)
>       (setq scroll-conservatively 101) ;; <<<<< this line was added
>       (cl-dotimes (i (* l 2))
>                   (insert "foo\n"))
>       (goto-char (point-min))
>       (forward-line (+ l 2))
>       (setq o (make-overlay (point) (point)))
>       (overlay-put o 'after-string "\n")
>       (goto-char (point-max))
>       (recenter -1))
>
>   . M-x eval-region RET
>
> After this, buffer "*Bug Repro*" pops up, with point at EOB positioned
> on the 2nd line from bottom of the window (instead of the last line).
>
> If you do the above, do you still get the cursor recentered in the
> window?  If so, I will try to play with display defaults in order to
> reproduce what you see.
>
> Thanks.
>

[-- Attachment #2: Type: text/html, Size: 2828 bytes --]

bug#23098: 25.0.92; (recenter -1) can leave point in the middle of the window

[Eli Zaretskii]

> From: Jorgen Schäfer <jorgen.schaefer@gmail.com>
> Date: Thu, 24 Mar 2016 16:27:53 +0000
> Cc: 23098@debbugs.gnu.org
> 
> > If you do the above, do you still get the cursor recentered in the
> > window?
> 
> Hm. That seems to work.

OK, so at least we see the same behavior.

> I was using let-binding around recenter, which does not work.

Indeed, it won't work because the value of scroll-conservatively needs
to be seen by redisplay, which runs after the code of the test case
finishes execution.  You could try let-binding it around a call to
'redisplay', but that's probably gross.

>  Even resetting
> scroll-conservatively after the call to recenter will exhibit the bug again. So setting scroll-conservatively is a
> possible workaround, although it means that *all* scrolling will be affected by scroll-conservatively. E.g.
> scrolling through the backlog of a M-x shell session would be affected, even though it just should affect the
> scrolling when the last line of the buffer is visible at all.

Well, setting scroll-conservatively is the only way in Emacs to
reliably avoid recentering of point in all kinds of corner cases.  So
if you cannot use that setting, I'm afraid I'm out of ideas.

Thanks.

bug#23098: 25.0.92; (recenter -1) can leave point in the middle of the window

[John Wiegley]

>>>>> Jorgen Schäfer <jorgen.schaefer@gmail.com> writes:

> The use case here are buffers like comint or IRC chat buffers.

FWIW, I use "scroll to bottom" in ERC, and all I had to do to make it work was
the following:

  (add-to-list
   'erc-mode-hook
   #'(lambda () (set (make-local-variable 'scroll-conservatively) 100)))

So is the bug here that it can't be done, or that (recenter -1) by itself
doesn't do what you think it should?

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

bug#23098: 25.0.92; (recenter -1) can leave point in the middle of the window

[Andreas Schwab]

John Wiegley <jwiegley@gmail.com> writes:

>   (add-to-list
>    'erc-mode-hook
>    #'(lambda () (set (make-local-variable 'scroll-conservatively) 100)))

You should use add-hook for hook variables.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

bug#23098: 25.0.92; (recenter -1) can leave point in the middle of the window

[Drew Adams]

> >   (add-to-list
> >    'erc-mode-hook
> >    #'(lambda () (set (make-local-variable 'scroll-conservatively) 100)))
> 
> You should use add-hook for hook variables.

Also (nits):

1. No need to use #' with lambdas (no advantage in a context that
   expects a function).

2. It is generally a good idea to use a named function (not a
   lambda) on a hook.  For one thing, removing it is easier and
   more robust (`remove-hook') - no worries about differences in
   whitespace etc.

3. Emacs now has `setq-local': `(setq-local VAR VALUE)'.  (I too
   use `(set (make-local-variable VAR) VALUE)', but mainly because
   I usually need code that works also for older Emacs versions.)

bug#23098: 25.0.92; (recenter -1) can leave point in the middle of the window

[Jorgen Schäfer]

[-- Attachment #1: Type: text/plain, Size: 1559 bytes --]

On Sun, Mar 27, 2016 at 12:33 AM John Wiegley <jwiegley@gmail.com> wrote:

> So is the bug here that it can't be done, or that (recenter -1) by itself
> doesn't do what you think it should?


As an end user, I expect `recenter-to-bottom' to cycle the line with point
on it between the top, middle and bottom of the current window. The problem
described in this report means that Emacs does not actually do this – the
line with point on it does not ever go even near the bottom of the window.

There is a technical reason for this problem: Emacs can not, actually,
display a multi-line overlay only partly. This is a shortcoming of Emacs,
but I assume this is non-trivial to fix. When Emacs notices it can not
fulfill a scroll request, it will use a fall back algorithm.

When I, as a user, try to display a line at the bottom of a window, it
would appear to me that trying to keep the line as close to the bottom as
possible instead of the middle of the window is a better fall back.

Setting scroll-conservatively to a large value is a way for the user to
work around this behavior. But this workaround also affects all the other
times a user tries to scroll the buffer. So this is not a fix, it's a
workaround with unintended and rather large side effects completely
unrelated to this.

A similar problems exist for programmers trying to implement a specific
behavior in buffers without prescribing to users how their normal scrolling
behaves.

This situation strikes me as an issue Emacs could address.

Regards,
Jorgen

[-- Attachment #2: Type: text/html, Size: 1954 bytes --]

bug#23098: 25.0.92; (recenter -1) can leave point in the middle of the window

[Eli Zaretskii]

> From: Jorgen Schäfer <jorgen.schaefer@gmail.com>
> Date: Mon, 28 Mar 2016 13:10:23 +0000
> Cc: 23098@debbugs.gnu.org
> 
> As an end user, I expect `recenter-to-bottom' to cycle the line with point on it between the top, middle and
> bottom of the current window. The problem described in this report means that Emacs does not actually do
> this – the line with point on it does not ever go even near the bottom of the window.

You are judging what Emacs does by the end result: since the result is
not to have point near the bottom, you deduce that Emacs didn't even
try.

But that is false.  What actually happens is that Emacs did try to do
what (recenter -1) requested, except that the display engine found
that doing so would have ended up with point being outside of the
window.  Since this is not acceptable, the display engine gave up and
recentered point in the window instead.

> There is a technical reason for this problem: Emacs can not, actually, display a multi-line overlay only partly.
> This is a shortcoming of Emacs, but I assume this is non-trivial to fix. When Emacs notices it can not fulfill a
> scroll request, it will use a fall back algorithm.

Yes.

> When I, as a user, try to display a line at the bottom of a window, it would appear to me that trying to keep the
> line as close to the bottom as possible instead of the middle of the window is a better fall back.

The problem is discovered at redisplay time, by some generic code that
has no direct relation to recentering, it gets run any time some
command requests that display of a window starts at a specified
position.  The fallback you'd like is more expensive, so using it in
that code would punish users who don't bump into this use case and
don't care about it.  Setting scroll-conservatively is the way to tell
Emacs you are generally prepared to deal with more expensive
(i.e. slower) redisplay if that gives you more accurate scrolling.
That's why, when scroll-conservatively is set to a large value, the
display engine does try a more accurate (and more expensive) fallback.
But doing that always, without the user's say-so, would be a
misfeature, I think.

> Setting scroll-conservatively to a large value is a way for the user to work around this behavior. But this
> workaround also affects all the other times a user tries to scroll the buffer. So this is not a fix, it's a
> workaround with unintended and rather large side effects completely unrelated to this.

I actually don't understand why you don't want to set
scroll-conservatively globally.  From what you say, it sounds like
that variable is exactly what you want, and not only in this
situation.  Can you explain why you don't see that as a solution?

> A similar problems exist for programmers trying to implement a specific behavior in buffers without
> prescribing to users how their normal scrolling behaves.

The idea is that users who are bothered by such use cases are expected
to like the effect of scroll-conservatively everywhere.

> This situation strikes me as an issue Emacs could address.

I'd welcome patches to try to fix this, but I think the best fix is
inside 'recenter': it should detect this situation and behave as if
its argument was modified so as not to request redisplay of the window
with window-start point that will cause recentering of point.  For
example, in the test case you presented, 'recenter' should behave as
if it was called with an argument of -2, not -1.

The tricky part is to implement a strategy that will do reasonable
things when the overlay or display string takes more than one line:
depending on the original value of the argument to 'recenter',
sometimes you'd want it increased, and sometimes decreased.
Interested?

bug#23098: 25.0.92; (recenter -1) can leave point in the middle of the window

[Jorgen Schäfer]

[-- Attachment #1: Type: text/plain, Size: 2132 bytes --]

On Mon, Mar 28, 2016 at 7:29 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Jorgen Schäfer <jorgen.schaefer@gmail.com>
> > Date: Mon, 28 Mar 2016 13:10:23 +0000
> > Cc: 23098@debbugs.gnu.org
> >
> > As an end user, I expect `recenter-to-bottom' to cycle the line with
> point on it between the top, middle and
> > bottom of the current window. The problem described in this report means
> that Emacs does not actually do
> > this – the line with point on it does not ever go even near the bottom
> of the window.
>
> You are judging what Emacs does by the end result: since the result is
> not to have point near the bottom, you deduce that Emacs didn't even
> try.
>

No, I am not. As I explained at length in my last mail.

I actually don't understand why you don't want to set
> scroll-conservatively globally.  From what you say, it sounds like
> that variable is exactly what you want, and not only in this
> situation.  Can you explain why you don't see that as a solution?
>

When scrolling through the buffer with cursor-up/cursor-down, I want
full-page scroll, not single-line scroll.

When I hit C-l, I want point to end up at the bottom at the window.

I do not want to lose the former just so I can have the latter.

And, as the programmer of a mode who tries to keep text at the bottom of
the window, I most certainly do not want to prescribe to the user how their
scrolling with cursor-up/cursor-down behaves just so I can keep the text at
the bottom of the window when I add a line.

These two situations have absolutely nothing to do with each other as far
as the user experience is concerned. The reason why the two interact is
purely technical and an artifact of how Emacs works underneath.

I'd welcome patches to try to fix this, but I think the best fix is
> inside 'recenter': it should detect this situation and behave as if
> its argument was modified so as not to request redisplay of the window
> with window-start point that will cause recentering of point.


Yes, for example. Great idea.


> Interested?
>

No, thank you.

Regards,
Jorgen

[-- Attachment #2: Type: text/html, Size: 3168 bytes --]

bug#23098: 25.0.92; (recenter -1) can leave point in the middle of the window

[Eli Zaretskii]

> From: Jorgen Schäfer <jorgen.schaefer@gmail.com>
> Date: Mon, 28 Mar 2016 17:41:02 +0000
> Cc: jwiegley@gmail.com, 23098@debbugs.gnu.org
> 
>  I actually don't understand why you don't want to set
>  scroll-conservatively globally. From what you say, it sounds like
>  that variable is exactly what you want, and not only in this
>  situation. Can you explain why you don't see that as a solution?
> 
> When scrolling through the buffer with cursor-up/cursor-down, I want full-page scroll, not single-line scroll.

Emacs doesn't by default give you a full-page scroll, it recenters
point instead.

> These two situations have absolutely nothing to do with each other as far as the user experience is
> concerned. The reason why the two interact is purely technical and an artifact of how Emacs works
> underneath.

No, the reason is not technical.  This was coded specifically for
those who do want minimal scroll to bring point into view.  They
explicitly requested this behavior.

>  I'd welcome patches to try to fix this, but I think the best fix is
>  inside 'recenter': it should detect this situation and behave as if
>  its argument was modified so as not to request redisplay of the window
>  with window-start point that will cause recentering of point. 
> 
> Yes, for example. Great idea.
> 
>  Interested?
> 
> No, thank you.

Well, I hope someone else will be.