bug#66942: move-to-column behavior when newlines have a display property

bug#66942: move-to-column behavior when newlines have a display property

[Sebastian Wålinder]

Hello! 

There was a bug that I encountered in a downstream Emacs package relating to setting line-move-visual to nil and running line-move when the package has created overlays on newlines.

The maintainer of the package has tracked down the bug to some incorrect behavior in the function move-to-column when overlays are present on newlines.

Here's a link to the issue that describes the issue with move-to-column:
https://github.com/jdtsmith/indent-bars/issues/22

Thoughts?

Thank you!

bug#66942: move-to-column behavior when newlines have a display property

[Eli Zaretskii]

unarchive 64988
forcemerge 66942 64988
thanks

> From: Sebastian Wålinder <s.walinder@gmail.com>
> Date: Sun, 05 Nov 2023 09:14:05 +1100
> 
> There was a bug that I encountered in a downstream Emacs package relating to setting line-move-visual to nil and running line-move when the package has created overlays on newlines.
> 
> The maintainer of the package has tracked down the bug to some incorrect behavior in the function move-to-column when overlays are present on newlines.
> 
> Here's a link to the issue that describes the issue with move-to-column:
> https://github.com/jdtsmith/indent-bars/issues/22
> 
> Thoughts?

It's the intended behavior.  It is the best Emacs can do given the
limitations posed by such overlay strings, see the detailed
explanation in bug#64988.

My best recommendation is not to use overlay strings with embedded
newlines, if accurate movement by columns is important.

bug#66942: move-to-column behavior when newlines have a display property

[JD Smith]

Eli Zaretskii <eliz@gnu.org> writes:

> ...unless someone comes with an idea for how to handle these situations in a sensible way (and I thought long and hard about it, but couldn't find such ideas), this will remain a limitation of move-to-column, and one of the complications introduced by display strings with embedded newlines in general.

Would it not make more sense to place the cursor _before_ the first character with a `\n` in its replacing-display property, if move-to-column attempts to move past it?  It is true that neither that position nor a position on a subsequent line is “correct”, but the former at least preserves the expected move-to-column semantics of “stay on this line”.  

The final newline is the only place to add display on blank lines, so this will be a general problem for any package attempting to do that.  Here’s a simple test of this for people wanting to investigate:

(progn
  (insert "\n>>>>\n" (propertize "\n" 'display "xxxxx\n") "SHOULD NOT MOVE HERE\n<<<<")
  (forward-line -2) ; at beginning of xxxxx\n line
  (move-to-column 10)) ; should not move past the line

bug#66942: move-to-column behavior when newlines have a display property

[Eli Zaretskii]

> From: JD Smith <jdtsmith@gmail.com>
> Date: Sun, 5 Nov 2023 12:18:12 -0500
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > ...unless someone comes with an idea for how to handle these situations in a sensible way (and I thought long and hard about it, but couldn't find such ideas), this will remain a limitation of move-to-column, and one of the complications introduced by display strings with embedded newlines in general.
> 
> Would it not make more sense to place the cursor _before_ the first character with a `\n` in its replacing-display property, if move-to-column attempts to move past it?  It is true that neither that position nor a position on a subsequent line is “correct”, but the former at least preserves the expected move-to-column semantics of “stay on this line”.  

"Stay on this line" has one meaning when the text is displayed in some
window and move-to-column is used for positioning the cursors, and
another meaning when move-to-column is used as part of a Lisp program
where move-to-column is used to move point (where "this line" is
generally the line of buffer text).

In addition, what you suggest will cause the behavior to change
drastically when a newline is added to the display property.

Moreover, did you also consider what should current-column return in
those cases?  These two must work in sync, as much as possible.

And finally, when neither behavior is 100% correct, I think we should
generally prefer the behavior we had for some time, because any
changes in that could break someone's code.  Whereas if someone wants
the behavior you describe, they can have that for the price of
relatively simple changes in their Lisp programs.