bug#64988: 30.0.50; move-to-column can move across lines if there is a text with display property

bug#64988: 30.0.50; move-to-column can move across lines if there is a text with display property

[Herman, Géza]

If there is a text with display property in a buffer, then 
move-to-column can move across lines.

Repro:
* emacs -Q
* in the scratch buffer, move the cursor to the top, and put an empty 
line at the beginning of the buffer
* execute 'M-: (put-text-property 1 2 'display "line\n")' (note: it's 
likely that the "\n" causes the problem)
* notice that the empty line becomes "line"
* while the cursor still on the first line, execute 'M-: (move-to-column 
20)'

The last command will move the cursor to the next line at column 16, 
instead of staying at the first line.

Note: I noticed this problem while using this package: 
https://github.com/jdtsmith/indent-bars.

The problem doesn't happen with emacs 28, this is the commit that 
introduced the issue:

4243747b1b8c3b7e3463822804b32e83febe2878 Fix 'current-column' in the 
presence of display strings

bug#64988: 30.0.50; move-to-column can move across lines if there is a text with display property

[Eli Zaretskii]

tags 64988 notabug wontfix
thanks

> Date: Tue, 1 Aug 2023 12:53:57 +0200
> From: Herman@debbugs.gnu.org, Géza <geza.herman@gmail.com>
> 
> If there is a text with display property in a buffer, then 
> move-to-column can move across lines.
> 
> Repro:
> * emacs -Q
> * in the scratch buffer, move the cursor to the top, and put an empty 
> line at the beginning of the buffer
> * execute 'M-: (put-text-property 1 2 'display "line\n")' (note: it's 
> likely that the "\n" causes the problem)
> * notice that the empty line becomes "line"
> * while the cursor still on the first line, execute 'M-: (move-to-column 
> 20)'
> 
> The last command will move the cursor to the next line at column 16, 
> instead of staying at the first line.
> 
> Note: I noticed this problem while using this package: 
> https://github.com/jdtsmith/indent-bars.
> 
> The problem doesn't happen with emacs 28, this is the commit that 
> introduced the issue:
> 
> 4243747b1b8c3b7e3463822804b32e83febe2878 Fix 'current-column' in the 
> presence of display strings

This is a known issue, but it is not a bug, at least not one we know
how to "fix".  In general, move-to-column cannot work correctly when
display strings with embedded newlines are involved, because Emacs
cannot place the cursor at the newline in the middle of such a display
string.

The commit to which you point fixed current-column to correctly report
the column where such display strings are involved, at least in the
important cases.  That move-to-column changed behavior in those cases
is unfortunate, but I couldn't find any sensible way of dealing with
these cases, because the correct result -- setting the cursor after
the newline embedded in the display string, because column 20 doesn't
exist on that line -- is impossible.  (The result that you expected,
which was what Emacs 28 did, i.e. for the cursor to stay at buffer
position 1, is also incorrect, since that position is column 1, not
column 20 and not the last column visible on that line.)

So I'm sorry, but unless someone comes with an idea for how to handle
these situations in a sensible way (and I thought long and hard about
it, but couldn't find such ideas), this will remain a limitation of
move-to-column, and one of the complications introduced by display
strings with embedded newlines in general.  The affected packages will
have to adapt to this change in some ways.  Since the original code
also produced incorrect results, just different incorrect results, I
don't see this as a serious problem, just as a bug that wasn't fixed,
but changed its buggy behavior.

bug#64988: 30.0.50; move-to-column can move across lines if there is a text with display property

[Herman, Geza]

I think, in my case, the move-to-column should completely ignore the 
display property. Is there a function which does that?

I noticed this problem because how indent-bars interacts with 
column-enforce-mode (https://github.com/jordonbiondo/column-enforce-mode).

column-enforce-mode uses move-to-column to mark long lines. And because 
the display property with "\n" basically joins neighboring lines, 
column-enforce-mode marks lines incorrectly.

I think it should only consider what's in the buffer, and ignore all 
rendering related properties. Is there a way to accomplish this?

Regarding how to fix this problem: wouldn't it make sense to stop 
calculating the width at the first "\n" in the displayed string? At 
least, it would fix this problem, as far as I understand. Not sure if it 
breaks anything. But supposedly nothing should depend on the exact 
behavior of column calculations, if the displayed string contains 
characters after the first "\n" (because it's kind of undefined what the 
width of a multi-line string is)

On 8/1/23 14:25, Eli Zaretskii wrote:
> tags 64988 notabug wontfix
> thanks
>
>> Date: Tue, 1 Aug 2023 12:53:57 +0200
>> From: Herman@debbugs.gnu.org, Géza <geza.herman@gmail.com>
>>
>> If there is a text with display property in a buffer, then
>> move-to-column can move across lines.
>>
>> Repro:
>> * emacs -Q
>> * in the scratch buffer, move the cursor to the top, and put an empty
>> line at the beginning of the buffer
>> * execute 'M-: (put-text-property 1 2 'display "line\n")' (note: it's
>> likely that the "\n" causes the problem)
>> * notice that the empty line becomes "line"
>> * while the cursor still on the first line, execute 'M-: (move-to-column
>> 20)'
>>
>> The last command will move the cursor to the next line at column 16,
>> instead of staying at the first line.
>>
>> Note: I noticed this problem while using this package:
>> https://github.com/jdtsmith/indent-bars.
>>
>> The problem doesn't happen with emacs 28, this is the commit that
>> introduced the issue:
>>
>> 4243747b1b8c3b7e3463822804b32e83febe2878 Fix 'current-column' in the
>> presence of display strings
> This is a known issue, but it is not a bug, at least not one we know
> how to "fix".  In general, move-to-column cannot work correctly when
> display strings with embedded newlines are involved, because Emacs
> cannot place the cursor at the newline in the middle of such a display
> string.
>
> The commit to which you point fixed current-column to correctly report
> the column where such display strings are involved, at least in the
> important cases.  That move-to-column changed behavior in those cases
> is unfortunate, but I couldn't find any sensible way of dealing with
> these cases, because the correct result -- setting the cursor after
> the newline embedded in the display string, because column 20 doesn't
> exist on that line -- is impossible.  (The result that you expected,
> which was what Emacs 28 did, i.e. for the cursor to stay at buffer
> position 1, is also incorrect, since that position is column 1, not
> column 20 and not the last column visible on that line.)
>
> So I'm sorry, but unless someone comes with an idea for how to handle
> these situations in a sensible way (and I thought long and hard about
> it, but couldn't find such ideas), this will remain a limitation of
> move-to-column, and one of the complications introduced by display
> strings with embedded newlines in general.  The affected packages will
> have to adapt to this change in some ways.  Since the original code
> also produced incorrect results, just different incorrect results, I
> don't see this as a serious problem, just as a bug that wasn't fixed,
> but changed its buggy behavior.

bug#64988: 30.0.50; move-to-column can move across lines if there is a text with display property

[Eli Zaretskii]

> Date: Tue, 1 Aug 2023 14:57:29 +0200
> Cc: 64988@debbugs.gnu.org
> From: "Herman, Geza" <geza.herman@gmail.com>
> 
> I think, in my case, the move-to-column should completely ignore the 
> display property. Is there a function which does that?
> 
> I noticed this problem because how indent-bars interacts with 
> column-enforce-mode (https://github.com/jordonbiondo/column-enforce-mode).
> 
> column-enforce-mode uses move-to-column to mark long lines. And because 
> the display property with "\n" basically joins neighboring lines, 
> column-enforce-mode marks lines incorrectly.
> 
> I think it should only consider what's in the buffer, and ignore all 
> rendering related properties. Is there a way to accomplish this?

Copy text into a temporary buffer, remove 'display' properties, call
move-to-column there, then move to the same position in the original
buffer?

> Regarding how to fix this problem: wouldn't it make sense to stop 
> calculating the width at the first "\n" in the displayed string?

Stop calculating and do what?  move-to-column doesn't just calculate,
it should move point to some place.  Where do you want to move point
in these cases?

> At least, it would fix this problem, as far as I understand. Not
> sure if it breaks anything. But supposedly nothing should depend on
> the exact behavior of column calculations, if the displayed string
> contains characters after the first "\n" (because it's kind of
> undefined what the width of a multi-line string is)

When the current line is shorter than N columns, move-to-column with
the argument of N should stop at the end of the line.  But it's
impossible to do that when the newline comes from a display property.
So the correct behavior is not "undefined" in this case, it is just
impossible to achieve.

bug#64988: 30.0.50; move-to-column can move across lines if there is a text with display property

[Herman, Geza]

On 8/1/23 15:26, Eli Zaretskii wrote:
> Copy text into a temporary buffer, remove 'display' properties, call
> move-to-column there, then move to the same position in the original
> buffer?
Yep, that's a possible solution, but I feel that it is such a basic 
functionality, emacs should be able to solve this question without using 
such workarounds. But this is just an intuition, feel free to ignore it 
:) Column-enforce-mode can be already slow for large files, if it had to 
do this copy all the time it would be even slower.

>> Regarding how to fix this problem: wouldn't it make sense to stop
>> calculating the width at the first "\n" in the displayed string?
> Stop calculating and do what?  move-to-column doesn't just calculate,
> it should move point to some place.  Where do you want to move point
> in these cases?
I understand that solving this problem if we want to stick with the 
current idea of move-to-column (meaning that it has to consider display 
properties), is hard, or even we can say that there are no good 
solutions. But ideally, it should move the cursor to the closest 
possible location in the same visual line. As far as I understand the 
intention of this function.

But for my problem, I don't think that this is the way to go. 
Column-enforce-mode should calculate columns without considering 
properties. I'd be good if emacs gave an out-of-box solution for this 
(without using copying), but if it doesn't, it's also fine.

bug#64988: 30.0.50; move-to-column can move across lines if there is a text with display property

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

> tags 64988 notabug wontfix
> thanks
[...]
> So I'm sorry, but unless someone comes with an idea for how to handle
> these situations in a sensible way (and I thought long and hard about
> it, but couldn't find such ideas), this will remain a limitation of
> move-to-column, and one of the complications introduced by display
> strings with embedded newlines in general.  The affected packages will
> have to adapt to this change in some ways.  Since the original code
> also produced incorrect results, just different incorrect results, I
> don't see this as a serious problem, just as a bug that wasn't fixed,
> but changed its buggy behavior.

I'm therefore closing this bug report.