bug#21468: 24.5; When next-line (visual) crosses overlay with before-string="\n" point goes to column 0

bug#21468: 24.5; When next-line (visual) crosses overlay with before-string="\n" point goes to column 0

[Noam Postavsky]

Run

    emacs -Q next-line-fail.el -l next-line-fail.el

where next-line-fail.el contains

(let ((pos 17))
  (goto-char pos)
  (overlay-put (make-overlay pos (1+ pos)) 'before-string
               (propertize "\n" 'face 'highlight)))

The overlay will create a highlighted blank line below line 1.
Move point forward so it's on line 1, column X (where X != 0), then do
C-n (next-line).

Expected: point should be on line 2 column X.
Actual: point goes to line 2, but in column 0.  Note that hitting C-n
a second time goes to line 3 column X.

The problem does not happen with (setq line-move-visual nil).

I've reproduced this in GUI and terminal.

This is a simplification from a magit bug
https://github.com/magit/magit/issues/2094

bug#21468: 24.5; When next-line (visual) crosses overlay with before-string="\n" point goes to column 0

[Eli Zaretskii]

> Date: Sun, 13 Sep 2015 01:17:39 -0400
> From: Noam Postavsky <npostavs@users.sourceforge.net>
> 
> Run
> 
>     emacs -Q next-line-fail.el -l next-line-fail.el
> 
> where next-line-fail.el contains
> 
> (let ((pos 17))
>   (goto-char pos)
>   (overlay-put (make-overlay pos (1+ pos)) 'before-string
>                (propertize "\n" 'face 'highlight)))
> 
> The overlay will create a highlighted blank line below line 1.
> Move point forward so it's on line 1, column X (where X != 0), then do
> C-n (next-line).
> 
> Expected: point should be on line 2 column X.
> Actual: point goes to line 2, but in column 0.  Note that hitting C-n
> a second time goes to line 3 column X.

Fixed in commit 6514b30 on master.

Once again, I must respectfully request that Magit maintainers refrain
from these practices of (ab)using Emacs display features where simpler
alternatives are readily available.  The buffer displayed by Magit is
entirely ephemeral, i.e. does not come from any file, so a much
simpler way would be to arrange the text in that buffer as Magit
needs, instead of using overlays and display strings to show something
that is not in the buffer.

E.g., in this fragment, cited by the Magit issue:

  (ov rend (1+ rend) 'after-string
      (propertize (concat (propertize "\s" 'display '(space :height (1)))
      		          (propertize "\n" 'line-height t))
                  'face 'magit-diff-lines-boundary)))

why can't Magit simply change the buffer text to be what it wants to
display? why does it need to use an overlay string?  All those text
properties are supported on buffer text as well.  Overlay strings and
display strings with embedded newlines are especially painful wrt
vertical cursor motion; changing the buffer text instead to present
the same display makes vertical motion's job much simpler.

The Emacs display engine was not designed for too heavy use of these
features.  By over-using them, you make Magit much less stable than
necessary, certainly much less future-proof and consistent across
different Emacs releases.  Please don't.

bug#21468: 24.5; When next-line (visual) crosses overlay with before-string="\n" point goes to column 0

[Noam Postavsky]

On Sun, Sep 13, 2015 at 6:30 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> Fixed in commit 6514b30 on master.

Thanks, it works.

> why can't Magit simply change the buffer text to be what it wants to
> display? why does it need to use an overlay string?

In this particular case, the overlays are delimiting a "magit region"
(consisting of whole logical lines touched by Emacs' region), having
them as actual text in the buffer would interfere with point movement
even more.

Or at least, that's what I saw when I tried just now to do something
similar with text properties:

(with-current-buffer (get-buffer-create "*text property test*")
  (erase-buffer)
  (insert-file "next-line-fail.el")
  (let ((pos 65))
    (goto-char pos)
    (insert (propertize (concat (propertize "\s" 'display '(space :height (1)))
                        (propertize "\n" 'line-height t))
                        'face 'highlight))
    (pop-to-buffer (current-buffer))))

bug#21468: 24.5; When next-line (visual) crosses overlay with before-string="\n" point goes to column 0

[Eli Zaretskii]

> Date: Sun, 13 Sep 2015 13:20:04 -0400
> From: Noam Postavsky <npostavs@users.sourceforge.net>
> Cc: 21468-done@debbugs.gnu.org
> 
> > why can't Magit simply change the buffer text to be what it wants to
> > display? why does it need to use an overlay string?
> 
> In this particular case, the overlays are delimiting a "magit region"
> (consisting of whole logical lines touched by Emacs' region), having
> them as actual text in the buffer would interfere with point movement
> even more.

Sorry, I don't understand: how can buffer contents interfere with
cursor motion?

> Or at least, that's what I saw when I tried just now to do something
> similar with text properties:
> 
> (with-current-buffer (get-buffer-create "*text property test*")
>   (erase-buffer)
>   (insert-file "next-line-fail.el")
>   (let ((pos 65))
>     (goto-char pos)
>     (insert (propertize (concat (propertize "\s" 'display '(space :height (1)))
>                         (propertize "\n" 'line-height t))
>                         'face 'highlight))
>     (pop-to-buffer (current-buffer))))

Please elaborate: what problems do you wee in this example?

bug#21468: 24.5; When next-line (visual) crosses overlay with before-string="\n" point goes to column 0

[Noam Postavsky]

On Sun, Sep 13, 2015 at 3:50 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>>
>> (with-current-buffer (get-buffer-create "*text property test*")
>>   (erase-buffer)
>>   (insert-file "next-line-fail.el")
>>   (let ((pos 65))
>>     (goto-char pos)
>>     (insert (propertize (concat (propertize "\s" 'display '(space :height (1)))
>>                         (propertize "\n" 'line-height t))
>>                         'face 'highlight))
>>     (pop-to-buffer (current-buffer))))
>
> Please elaborate: what problems do you see in this example?

With point on line 1 (preceding the thin line), doing next-line ends
up inside the thin line, rather than after it.

bug#21468: 24.5; When next-line (visual) crosses overlay with before-string="\n" point goes to column 0

[Eli Zaretskii]

> Date: Sun, 13 Sep 2015 16:22:10 -0400
> From: Noam Postavsky <npostavs@users.sourceforge.net>
> Cc: 21468@debbugs.gnu.org
> 
> On Sun, Sep 13, 2015 at 3:50 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> >>
> >> (with-current-buffer (get-buffer-create "*text property test*")
> >>   (erase-buffer)
> >>   (insert-file "next-line-fail.el")
> >>   (let ((pos 65))
> >>     (goto-char pos)
> >>     (insert (propertize (concat (propertize "\s" 'display '(space :height (1)))
> >>                         (propertize "\n" 'line-height t))
> >>                         'face 'highlight))
> >>     (pop-to-buffer (current-buffer))))
> >
> > Please elaborate: what problems do you see in this example?
> 
> With point on line 1 (preceding the thin line), doing next-line ends
> up inside the thin line, rather than after it.

Of course: that's a line of text.  Why is that a problem?  (I have no
idea what role is that thin line serving in Magit.)

Do you see any other issues with cursor motion?  I don't.

bug#21468: 24.5; When next-line (visual) crosses overlay with before-string="\n" point goes to column 0

[Noam Postavsky]

On Mon, Sep 14, 2015 at 2:17 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>
> Of course: that's a line of text.  Why is that a problem?  (I have no
> idea what role is that thin line serving in Magit.)

In this particular case, the overlays are delimiting a "magit region"
(consisting of whole logical lines touched by Emacs' region). This
means the thin line has to move with point, staying just ahead of it.

Here is a demo, eval this in a buffer, then do set-mark-command and
move around a bit:

(require 'cl-lib)

(defvar-local magit-region-overlays nil)

(defun 21468-update-hunk-region (start end window rol)
  (mapc #'delete-overlay magit-region-overlays)
  (let ((rbeg (save-excursion (goto-char (region-beginning))
                              (line-beginning-position)))
        (rend (save-excursion (goto-char (region-end))
                              (line-end-position))))
    (cl-flet ((ov (start end &rest args)
                  (let ((ov (make-overlay start end nil t)))
                    (overlay-put ov 'evaporate t)
                    (while args (overlay-put ov (pop args) (pop args)))
                    (push ov magit-region-overlays)
                    ov)))
      (ov rbeg (1+ rbeg) 'before-string
          (propertize (concat (propertize "\s" 'display '(space :height (1)))
                              (propertize "\n" 'line-height t))
                      'face 'highlight))
      (ov rend (1+ rend) 'after-string
          (propertize (concat (propertize "\s" 'display '(space :height (1)))
                              (propertize "\n" 'line-height t))
                      'face 'highlight)))))

(setq-local redisplay-highlight-region-function '21468-update-hunk-region)

(setq-local redisplay-unhighlight-region-function
            (lambda (rol) (mapc #'delete-overlay magit-region-overlays)))

bug#21468: 24.5; When next-line (visual) crosses overlay with before-string="\n" point goes to column 0

[Stefan Monnier]

> In this particular case, the overlays are delimiting a "magit region"
> (consisting of whole logical lines touched by Emacs' region). This
> means the thin line has to move with point, staying just ahead of it.

While I think I can see why you'd do it that way in earlier Emacsen,
I think the better way to do it nowadays is to simply arrange for the
region highlighting to extend to to whole lines.

You can do that with redisplay-(un)highlight-region-function, which was
introduced to allow rectangular-region-highlighting.


        Stefan

bug#21468: 24.5; When next-line (visual) crosses overlay with before-string="\n" point goes to column 0

[Eli Zaretskii]

> Date: Mon, 14 Sep 2015 08:46:57 -0400
> From: Noam Postavsky <npostavs@users.sourceforge.net>
> Cc: 21468@debbugs.gnu.org
> 
> On Mon, Sep 14, 2015 at 2:17 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > Of course: that's a line of text.  Why is that a problem?  (I have no
> > idea what role is that thin line serving in Magit.)
> 
> In this particular case, the overlays are delimiting a "magit region"
> (consisting of whole logical lines touched by Emacs' region). This
> means the thin line has to move with point, staying just ahead of it.

What's wrong with marking the region with some special background
color?

Anyway, if you must use overlays here, by all means do, just try to
avoid newlines in its strings, if you want reliable cursor movement
across that overlay.

Please understand: what you originally perceived as a "bug" was
actually normal Emacs routine of placing point immediately after the
end of the overlay string.  It ended up in column zero because the
overlay string ended in a newline.  If there were no newline, the
problem would not have happened.  We now have special code in
vertical-motion that caters to this specific scenario.  That's why
using such overlay strings should be discouraged: they tend to force
us add similar special-case code all over the place.

> Here is a demo, eval this in a buffer, then do set-mark-command and
> move around a bit:

What am I supposed to see?  I see nothing that I don't see without
this.

bug#21468: 24.5; When next-line (visual) crosses overlay with before-string="\n" point goes to column 0

[Noam Postavsky]

On Mon, Sep 14, 2015 at 9:33 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> What's wrong with marking the region with some special background
> color?

I saw the fix you applied, and I certainly understand your distaste at
adding these special cases, but I don't want to start debating the
aesthetics now.

>
> What am I supposed to see?  I see nothing that I don't see without
> this.

In that buffer, after evaluating its code, when region is active you
should see 2 highlighted thin lines delimiting the region, instead of
the normal region highlight.

bug#21468: 24.5; When next-line (visual) crosses overlay with before-string="\n" point goes to column 0

[Eli Zaretskii]

> Date: Mon, 14 Sep 2015 09:55:57 -0400
> From: Noam Postavsky <npostavs@users.sourceforge.net>
> Cc: 21468@debbugs.gnu.org
> 
> On Mon, Sep 14, 2015 at 9:33 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> > What's wrong with marking the region with some special background
> > color?
> 
> I saw the fix you applied, and I certainly understand your distaste at
> adding these special cases, but I don't want to start debating the
> aesthetics now.

It's not the aesthetics I'm worried about, it's maintainability.  I
had the dubious pleasure of writing most of those special-case parts,
and I'm already beginning to have difficulty remembering what use case
does each one cater to, and how to make sure I don't break it by some
change.  How can we expect others to make changes in such a mess?

> > What am I supposed to see?  I see nothing that I don't see without
> > this.
> 
> In that buffer, after evaluating its code, when region is active you
> should see 2 highlighted thin lines delimiting the region, instead of
> the normal region highlight.

I think Stefan suggested a way to achieve this without all this
complexity.  I hope it will be possible.

bug#21468: 24.5; When next-line (visual) crosses overlay with before-string="\n" point goes to column 0

[Noam Postavsky]

On Mon, Sep 14, 2015 at 9:22 AM, Stefan Monnier
<monnier@iro.umontreal.ca> wrote:
> I think the better way to do it nowadays is to simply arrange for the
> region highlighting to extend to to whole lines.

On Mon, Sep 14, 2015 at 11:14 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Mon, 14 Sep 2015 09:55:57 -0400
>> From: Noam Postavsky <npostavs@users.sourceforge.net>
>> In that buffer, after evaluating its code, when region is active you
>> should see 2 highlighted thin lines delimiting the region, instead of
>> the normal region highlight.
>
> I think Stefan suggested a way to achieve this without all this
> complexity.  I hope it will be possible.

If you can suggest a way to get horizontal bars that are ignored by
point movement without using newlines in overlays, then great. I
almost got there using :overline and :underline, but as far as I can
tell, there's no way to extend the line to the edge of the screen
without using an overlay with a newline.

If you just want to say that the region should be shown by
highlighting instead of horizontal bars, then *shrug*.

bug#21468: 24.5; When next-line (visual) crosses overlay with before-string="\n" point goes to column 0

[Eli Zaretskii]

> Date: Mon, 14 Sep 2015 16:50:10 -0400
> From: Noam Postavsky <npostavs@users.sourceforge.net>
> Cc: 21468@debbugs.gnu.org
> 
> If you can suggest a way to get horizontal bars that are ignored by
> point movement without using newlines in overlays, then great. I
> almost got there using :overline and :underline, but as far as I can
> tell, there's no way to extend the line to the edge of the screen
> without using an overlay with a newline.

I think you should be able to extend the face by using a display
property with :align-to value.  The value should be computed to go all
the way to the edge of the window.

> If you just want to say that the region should be shown by
> highlighting instead of horizontal bars, then *shrug*.

I cannot do anything about these issues besides explaining how they
harm Emacs maintenance, and asking people to look for alternative
solutions.  And yes, highlighting sounds like a good alternative to
me.  Feel free to ignore the requests, if you don't consider them
important enough.

bug#21468: 24.5; When next-line (visual) crosses overlay with before-string="\n" point goes to column 0

[Noam Postavsky]

[-- Attachment #1: Type: text/plain, Size: 1283 bytes --]

On Tue, Sep 15, 2015 at 2:53 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>
> I think you should be able to extend the face by using a display
> property with :align-to value.  The value should be computed to go all
> the way to the edge of the window.

We tried something like this:

(let ((align `(space :align-to (+ (,(window-body-width nil t))
                                  ,(window-hscroll)))))
  (ov rbeg eol 'face (list :overline color)
      'after-string (propertize "\s" 'face face 'display align)))

But when moving point to end of line this causes the cursor to appear
at the edge of the window instead of at the "real" end of line. I
guess it's because the aligned space pushes the newline character to
the edge of the window.

For a self-contained example do "emacs -Q overlay.el -f eval-buffer -f
set-mark-command -f move-end-of-line" with attached overlay.el.


> I cannot do anything about these issues besides explaining how they
> harm Emacs maintenance, and asking people to look for alternative
> solutions.

Well you *could* refuse to fix bugs like this, i.e. declare that
creating overlays with newlines invokes undefined behaviour. As it
stands, the best way to get the effect we want is to use an overlayed
newline, especially now that you've fixed this bug.

[-- Attachment #2: overlay.el --]
[-- Type: application/octet-stream, Size: 1763 bytes --]

(require 'cl-lib)

(defvar-local magit-region-overlays nil)

(defun 21468-update-hunk-region (start end window rol)
  (mapc #'delete-overlay magit-region-overlays)
  (cl-flet ((ov (start end &rest args)
                (let ((ov (make-overlay start end nil t)))
                  (while args (overlay-put ov (pop args) (pop args)))
                  (push ov magit-region-overlays)
                  ov)))
    (let* ((align (list 'space :align-to `(+ (,(window-body-width nil t))
                                             ,(window-hscroll))))
           (rbeg-line (save-excursion (goto-char start)
                                      (cons (line-beginning-position)
                                            (line-end-position))))
           (rend-line (save-excursion (goto-char end)
                                      (cons (line-beginning-position)
                                            (line-end-position))))
           (color (face-background 'highlight nil t))
           (face  (list :overline color :underline color)))
      (if (equal rbeg-line rend-line)
          (ov (car rbeg-line) (cdr rbeg-line) 'face face
              'after-string (propertize "\s" 'face face 'display align))
        (ov (car rbeg-line) (cdr rbeg-line) 'face (setq face (list :overline color))
            'after-string (propertize "\s" 'face face 'display align))
        (ov (car rend-line) (cdr rend-line) 'face (setq face (list :underline color))
            'after-string (propertize "\s" 'face face 'display align))))))

(setq-local redisplay-highlight-region-function '21468-update-hunk-region)

(setq-local redisplay-unhighlight-region-function
            (lambda (rol) (mapc #'delete-overlay magit-region-overlays)))

bug#21468: 24.5; When next-line (visual) crosses overlay with before-string="\n" point goes to column 0

[Eli Zaretskii]

> Date: Thu, 1 Oct 2015 16:43:11 -0400
> From: Noam Postavsky <npostavs@users.sourceforge.net>
> Cc: 21468@debbugs.gnu.org
> 
> > I think you should be able to extend the face by using a display
> > property with :align-to value.  The value should be computed to go all
> > the way to the edge of the window.
> 
> We tried something like this:
> 
> (let ((align `(space :align-to (+ (,(window-body-width nil t))
>                                   ,(window-hscroll)))))
>   (ov rbeg eol 'face (list :overline color)
>       'after-string (propertize "\s" 'face face 'display align)))
> 
> But when moving point to end of line this causes the cursor to appear
> at the edge of the window instead of at the "real" end of line. I
> guess it's because the aligned space pushes the newline character to
> the edge of the window.

Yes.  But why is that a problem?

If you dislike that, you could make the stretch one column shorter,
like this:

  (let* ((align (list 'space :align-to `(+ (,(- (window-body-width nil t)
						(default-font-width)))
					   ,(window-hscroll))))

(Btw, I think adding window-hscroll is incorrect here, as its value is
in columns, while window-body-width returns the width in pixels.)

> > I cannot do anything about these issues besides explaining how they
> > harm Emacs maintenance, and asking people to look for alternative
> > solutions.
> 
> Well you *could* refuse to fix bugs like this, i.e. declare that
> creating overlays with newlines invokes undefined behaviour.

I could, but I consider that a "doomsday weapon", hopefully never to
be used in Emacs development, certainly not by me.  I don't think it
was ever done.

bug#21468: 24.5; When next-line (visual) crosses overlay with before-string="\n" point goes to column 0

[Noam Postavsky]

On Fri, Oct 2, 2015 at 6:01 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>> But when moving point to end of line this causes the cursor to appear
>> at the edge of the window instead of at the "real" end of line. I
>> guess it's because the aligned space pushes the newline character to
>> the edge of the window.
>
> Yes.  But why is that a problem?

Because we want the lines to be just visual effects that don't affect
cursor movement, i.e. the cursor should act the same whether or not
the lines are visible.

>
> If you dislike that, you could make the stretch one column shorter,

Which of course just means that the cursor appears at one column
before the edge of the window instead of at the "real" end of line.

>   (let* ((align (list 'space :align-to `(+ (,(- (window-body-width nil t)
>                                                 (default-font-width)))
>                                            ,(window-hscroll))))
>
> (Btw, I think adding window-hscroll is incorrect here, as its value is
> in columns, while window-body-width returns the width in pixels.)

window-body-width is inside a list indicating pixel value, and
window-hscroll is a plain integer indicating a multiple of the font
width, so it should work, no? That was my reading of "37.16.3 Pixel
Specification for Spaces" in the elisp manual.

From: Noam Postavsky <npostavs@users.sourceforge.net>
> For a self-contained example do "emacs -Q overlay.el -f eval-buffer -f
> set-mark-command -f move-end-of-line" with attached overlay.el.

By the way, I noticed this is wrong (for some reason it works in 24.5
even though functions are being called with the wrong number of
arguments), it should be called like

    emacs -Q overlay.el --eval "(progn (switch-to-buffer
\"overlay.el\") (eval-buffer) (set-mark-command nil) (move-end-of-line
nil))"

bug#21468: 24.5; When next-line (visual) crosses overlay with before-string="\n" point goes to column 0

[Eli Zaretskii]

> Date: Fri, 2 Oct 2015 15:58:08 -0400
> From: Noam Postavsky <npostavs@users.sourceforge.net>
> Cc: 21468@debbugs.gnu.org
> 
> On Fri, Oct 2, 2015 at 6:01 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> >> But when moving point to end of line this causes the cursor to appear
> >> at the edge of the window instead of at the "real" end of line. I
> >> guess it's because the aligned space pushes the newline character to
> >> the edge of the window.
> >
> > Yes.  But why is that a problem?
> 
> Because we want the lines to be just visual effects that don't affect
> cursor movement, i.e. the cursor should act the same whether or not
> the lines are visible.
> 
> >
> > If you dislike that, you could make the stretch one column shorter,
> 
> Which of course just means that the cursor appears at one column
> before the edge of the window instead of at the "real" end of line.

If you want to put the cursor there, you need an overlay with a
'cursor' property.

Or we could have a new feature whereby the stretch of white space
would allow positioning the cursor on it.  Right now, the flag that
disallows positioning the cursor is hard-coded in the display engine.

bug#21468: 24.5; When next-line (visual) crosses overlay with before-string="\n" point goes to column 0

[Noam Postavsky]

On 10/2/15, Eli Zaretskii <eliz@gnu.org> wrote:
>
> If you want to put the cursor there, you need an overlay with a
> 'cursor' property.

This works nicely. But there are some other glitches.

- On a truncated line with the overlay, move-end-of-line skips to the
next line. (I tried to avoid this by only adding the stretched space
when it's going to be visible. But that just made things worse
(looping in the display engine))

- When at the end of line, next-line goes to the end-of-line instead
of staying on the same column.

> Or we could have a new feature whereby the stretch of white space
> would allow positioning the cursor on it.  Right now, the flag that
> disallows positioning the cursor is hard-coded in the display engine.

Could be interesting, but I don't know of any need for it at present.

bug#21468: 24.5; When next-line (visual) crosses overlay with before-string="\n" point goes to column 0

[Eli Zaretskii]

> Date: Fri, 9 Oct 2015 19:34:43 -0400
> From: Noam Postavsky <npostavs@users.sourceforge.net>
> Cc: 21468@debbugs.gnu.org
> 
> On 10/2/15, Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > If you want to put the cursor there, you need an overlay with a
> > 'cursor' property.
> 
> This works nicely. But there are some other glitches.
> 
> - On a truncated line with the overlay, move-end-of-line skips to the
> next line. (I tried to avoid this by only adding the stretched space
> when it's going to be visible. But that just made things worse
> (looping in the display engine))
> 
> - When at the end of line, next-line goes to the end-of-line instead
> of staying on the same column.

Are these caused by the 'cursor' property, i.e. do not happen if that
property is not used?

In any case, I'd appreciate a short self-contained test case for each
of these two issues, as I'm not sure I understand what's going on.

bug#21468: 24.5; When next-line (visual) crosses overlay with before-string="\n" point goes to column 0

[Noam Postavsky]

[-- Attachment #1: Type: text/plain, Size: 1257 bytes --]

On 10/10/15, Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Fri, 9 Oct 2015 19:34:43 -0400
>> From: Noam Postavsky <npostavs@users.sourceforge.net>
>> - On a truncated line with the overlay, move-end-of-line skips to the
>> next line. (I tried to avoid this by only adding the stretched space
>> when it's going to be visible. But that just made things worse
>> (looping in the display engine))
>>
>> - When at the end of line, next-line goes to the end-of-line instead
>> of staying on the same column.
>
> Are these caused by the 'cursor' property, i.e. do not happen if that
> property is not used?

They're not caused by the 'cursor' property, although it makes the 2nd
case more visually surprising.

>
> In any case, I'd appreciate a short self-contained test case for each
> of these two issues, as I'm not sure I understand what's going on.
>

Using attached 21468-overlay.el

The 1st case:

   emacs -Q -l 21468-overlay.el -f 21468-test-eol-skip

then hit C-e. Point lands at the end of line 19 instead of the end of
line 18 where it started.

The 2nd case

   emacs -Q -l 21468-overlay.el -f 21468-test-eol-col-movement

then hit C-e C-n. Point lands at the end of line 8 instead of line 8,
column 33 which is what happens when there is no overlay.

[-- Attachment #2: 21468-overlay.el --]
[-- Type: application/octet-stream, Size: 1065 bytes --]

(defconst +file-name+ load-file-name)
(defun 21468-put-overlay-at-line (line)
  (goto-line line)
  (let ((ov (make-overlay (line-beginning-position) (line-end-position)))
        (face (list :overline "red" :underline "green")))
    (overlay-put ov 'face face)
    (overlay-put ov 'after-string
                 (propertize "\s" 'face face
                             'display (list 'space :align-to
                                            `(+ (,(window-body-width nil t))
                                                ,(window-hscroll)))
                             'cursor t))))
(defun 21468-test-eol-skip ()
  (find-file +file-name+)
  (setq truncate-lines t)
  ;; line 18 is truncated (assuming window width < 200)
  (21468-put-overlay-at-line 18)
  ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
  )

(defun 21468-test-eol-col-movement ()
  (find-file +file-name+)
  (21468-put-overlay-at-line 7))

bug#21468: 24.5; When next-line (visual) crosses overlay with before-string="\n" point goes to column 0

[Eli Zaretskii]

> Date: Sat, 10 Oct 2015 09:37:43 -0400
> From: Noam Postavsky <npostavs@users.sourceforge.net>
> Cc: 21468@debbugs.gnu.org
> 
> The 2nd case
> 
>    emacs -Q -l 21468-overlay.el -f 21468-test-eol-col-movement
> 
> then hit C-e C-n. Point lands at the end of line 8 instead of line 8,
> column 33 which is what happens when there is no overlay.

This is not a bug: vertical-motion doesn't know about the 'cursor'
property, so it works as if the property didn't exist.

The 'cursor' property is only considered when actually putting the
cursor on screen.

I will look into the first case, sigh.

bug#21468: 24.5; When next-line (visual) crosses overlay with before-string="\n" point goes to column 0

[Noam Postavsky]

On Sat, Oct 10, 2015 at 10:20 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Sat, 10 Oct 2015 09:37:43 -0400
>> From: Noam Postavsky <npostavs@users.sourceforge.net>
>> Cc: 21468@debbugs.gnu.org
>>
>> The 2nd case
>>
>>    emacs -Q -l 21468-overlay.el -f 21468-test-eol-col-movement
>>
>> then hit C-e C-n. Point lands at the end of line 8 instead of line 8,
>> column 33 which is what happens when there is no overlay.
>
> This is not a bug: vertical-motion doesn't know about the 'cursor'
> property, so it works as if the property didn't exist.

Okay, that makes sense, but it means this approach is worse than
overlayed newlines for magit.

bug#21468: 24.5; When next-line (visual) crosses overlay with before-string="\n" point goes to column 0

[Eli Zaretskii]

> Date: Sat, 10 Oct 2015 09:37:43 -0400
> From: Noam Postavsky <npostavs@users.sourceforge.net>
> Cc: 21468@debbugs.gnu.org
> 
> >> - On a truncated line with the overlay, move-end-of-line skips to the
> >> next line. (I tried to avoid this by only adding the stretched space
> >> when it's going to be visible. But that just made things worse
> >> (looping in the display engine))
> >>
> >> - When at the end of line, next-line goes to the end-of-line instead
> >> of staying on the same column.
> >
> > Are these caused by the 'cursor' property, i.e. do not happen if that
> > property is not used?
> 
> They're not caused by the 'cursor' property, although it makes the 2nd
> case more visually surprising.
> 
> >
> > In any case, I'd appreciate a short self-contained test case for each
> > of these two issues, as I'm not sure I understand what's going on.
> >
> 
> Using attached 21468-overlay.el
> 
> The 1st case:
> 
>    emacs -Q -l 21468-overlay.el -f 21468-test-eol-skip
> 
> then hit C-e. Point lands at the end of line 19 instead of the end of
> line 18 where it started.

Fixed.

bug#21468: 24.5; When next-line (visual) crosses overlay with before-string="\n" point goes to column 0

[Eli Zaretskii]

> Date: Sat, 10 Oct 2015 12:51:06 -0400
> From: Noam Postavsky <npostavs@users.sourceforge.net>
> Cc: 21468@debbugs.gnu.org
> 
> On Sat, Oct 10, 2015 at 10:20 AM, Eli Zaretskii <eliz@gnu.org> wrote:
> >> Date: Sat, 10 Oct 2015 09:37:43 -0400
> >> From: Noam Postavsky <npostavs@users.sourceforge.net>
> >> Cc: 21468@debbugs.gnu.org
> >>
> >> The 2nd case
> >>
> >>    emacs -Q -l 21468-overlay.el -f 21468-test-eol-col-movement
> >>
> >> then hit C-e C-n. Point lands at the end of line 8 instead of line 8,
> >> column 33 which is what happens when there is no overlay.
> >
> > This is not a bug: vertical-motion doesn't know about the 'cursor'
> > property, so it works as if the property didn't exist.
> 
> Okay, that makes sense, but it means this approach is worse than
> overlayed newlines for magit.

I don't think so.  I think a casual mismatch in a column while moving
through a buffer, and in marginal cases at that, is not a big deal.

bug#21468: 24.5; When next-line (visual) crosses overlay with before-string="\n" point goes to column 0

[Noam Postavsky]

On Sat, Oct 10, 2015 at 1:09 PM, Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Sat, 10 Oct 2015 12:51:06 -0400
>> From: Noam Postavsky <npostavs@users.sourceforge.net>
>>
>> Okay, that makes sense, but it means this approach is worse than
>> overlayed newlines for magit.
>
> I don't think so.  I think a casual mismatch in a column while moving
> through a buffer, and in marginal cases at that, is not a big deal.

It's not a *big* deal, but with Emacs 25 using overlayed newlines
gives no problems at all.

bug#21468: 24.5; When next-line (visual) crosses overlay with before-string="\n" point goes to column 0

[Eli Zaretskii]

> Date: Sat, 10 Oct 2015 17:06:59 -0400
> From: Noam Postavsky <npostavs@users.sourceforge.net>
> Cc: 21468@debbugs.gnu.org
> 
> On Sat, Oct 10, 2015 at 1:09 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> >> Date: Sat, 10 Oct 2015 12:51:06 -0400
> >> From: Noam Postavsky <npostavs@users.sourceforge.net>
> >>
> >> Okay, that makes sense, but it means this approach is worse than
> >> overlayed newlines for magit.
> >
> > I don't think so.  I think a casual mismatch in a column while moving
> > through a buffer, and in marginal cases at that, is not a big deal.
> 
> It's not a *big* deal, but with Emacs 25 using overlayed newlines
> gives no problems at all.

I'm quite sure problems will surface sooner or later.  Overlays with
newlines hit on fragile parts of the display engine code, which was
never really designed for that kind of stuff.