bug#57433: Emacs no longer moves point into visible port of the buffer

bug#57433: Emacs no longer moves point into visible port of the buffer

[Paul Pogonyshev]

[-- Attachment #1: Type: text/plain, Size: 853 bytes --]

Git commit 233c91d32a. Has been broken at least for a few days: my previous
build also exhibited this.

No idea how to create a short test in Elisp.

From user point of view: open a Magit buffer in a directory with at least
two locally modified and uncommitted files. The buffer shows sth. like this:

...
Staged changes (N)
modified   file1
modified   file2
...

Move the point with C-n to the line with `file2'. Press TAB. Magit section
for the file is not opened. Use `M-x describe-char'. From the resulting
buffer you can say that the point is actually on an invisible char '@',
i.e. start of (hidden) diff for the first file.

It may be necessary to restart Emacs if you have used Magit in this
directory previously.

This used to work fine (i.e. point was never on an invisible character)
both in Emacs 28 and earlier snapshots of `master'.

Paul

[-- Attachment #2: Type: text/html, Size: 1109 bytes --]

bug#57433: Emacs no longer moves point into visible port of the buffer

[Gerd Möllmann]

Paul Pogonyshev <pogonyshev@gmail.com> writes:

> Git commit 233c91d32a. Has been broken at least for a few days: my previous build also exhibited this.
>
> No idea how to create a short test in Elisp.
>
> From user point of view: open a Magit buffer in a directory with at least two locally modified and uncommitted files. The buffer shows sth. like
> this:
>
> ...
> Staged changes (N)
> modified   file1
> modified   file2
> ...
>
> Move the point with C-n to the line with `file2'. Press TAB. Magit section for the file is not opened. Use `M-x describe-char'. From the resulting
> buffer you can say that the point is actually on an invisible char '@', i.e. start of (hidden) diff for the first file.
>
> It may be necessary to restart Emacs if you have used Magit in this directory previously.
>
> This used to work fine (i.e. point was never on an invisible character) both in Emacs 28 and earlier snapshots of `master'.
>
> Paul

I can reproduce this.  Bisecting.

bug#57433: Emacs no longer moves point into visible port of the buffer

[Gerd Möllmann]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

a71c05b44de74fe16691f680df34c4534992e472 is the first bad commit
commit a71c05b44de74fe16691f680df34c4534992e472
Author: Eli Zaretskii <eliz@gnu.org>
Date:   Sun Aug 14 15:47:59 2022 +0300
    Further speedups of redisplay of long and truncated lines
    
    * src/xdisp.c (mode_line_update_needed, redisplay_window)
    (decode_mode_spec): Don't avoid calling current_column, as it is
    now fast enough.
    (redisplay_window) <optional_new_start>: Don't call 'move_it_to'
    if its result will not be used.
    (Flong_line_optimizations_p): New primitive.
    * src/indent.c (Fcurrent_column): Doc fix.
    (current_column, scan_for_column): When in a buffer with long
    and/or truncated lines, quickly return an approximate value.
    * src/window.c (Frecenter): Use the old text-mode code when the
    buffer has very long lines.
    
    * lisp/simple.el (line-move): Avoid costly calls to
    'line-move-partial' and 'line-move-visual' when lines are
    truncated and/or very long.
    (move-beginning-of-line): Call 'line-beginning-position' instead
    of the slower 'skip-chars-backward'.
    
    * etc/NEWS: Announce 'long-line-optimizations-p'.
 etc/NEWS       |  3 +++
 lisp/simple.el | 31 ++++++++++++++++++++---
 src/indent.c   | 60 ++++++++++++++++++++++++++++++++++++++------
 src/window.c   |  9 ++++---
 src/xdisp.c    | 79 +++++++++++++++++++++++++++++-----------------------------
 5 files changed, 129 insertions(+), 53 deletions(-)


I will look at the changes in simple.el soon if no-one beats me.

bug#57433: Emacs no longer moves point into visible port of the buffer

[Gerd Möllmann]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>
> a71c05b44de74fe16691f680df34c4534992e472 is the first bad commit
>
> I will look at the changes in simple.el soon if no-one beats me.

Hm, I must admit that I don't understand the change in line-move, or
better said the context in which it was made.  So if I fix this I
probably break some optimization.

I have a simple recipe now, though.

emacs -Q

Insert and eval this function:

(defun hansi ()
  (insert "line 1\n")
  (insert (propertize "+invisible 1\ninvisible2-\n" 'invisible t))
  (insert "line 2\n"))

M-: (hansi) RET
C-p
C-x =
=>
Char: + (43, #o53, #x2b) point=330 of 365 (90%) column=0

That's something for Eli, I'd say.

bug#57433: Emacs no longer moves point into visible port of the buffer

[Eli Zaretskii]

> Cc: 57433@debbugs.gnu.org
> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Date: Sat, 27 Aug 2022 10:53:37 +0200
> 
> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
> 
> > Gerd Möllmann <gerd.moellmann@gmail.com> writes:
> >
> > a71c05b44de74fe16691f680df34c4534992e472 is the first bad commit
> >
> > I will look at the changes in simple.el soon if no-one beats me.
> 
> Hm, I must admit that I don't understand the change in line-move, or
> better said the context in which it was made.  So if I fix this I
> probably break some optimization.

The changes in line-move should only affect buffers with very long
lines.

> I have a simple recipe now, though.
> 
> emacs -Q
> 
> Insert and eval this function:
> 
> (defun hansi ()
>   (insert "line 1\n")
>   (insert (propertize "+invisible 1\ninvisible2-\n" 'invisible t))
>   (insert "line 2\n"))
> 
> M-: (hansi) RET
> C-p
> C-x =
> =>
> Char: + (43, #o53, #x2b) point=330 of 365 (90%) column=0

Why is that a problem?  I get the same result on the emacs-28 branch,
where the changes in this commit were not installed.  And the OP said
the problem is new on master.

bug#57433: Emacs no longer moves point into visible port of the buffer

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

>> Char: + (43, #o53, #x2b) point=330 of 365 (90%) column=0
>
> Why is that a problem?  I get the same result on the emacs-28 branch,
> where the changes in this commit were not installed.  And the OP said
> the problem is new on master.

Then it's probably not the right recipe, sorry.

bug#57433: Emacs no longer moves point into visible port of the buffer

[Eli Zaretskii]

> Cc: 57433@debbugs.gnu.org, pogonyshev@gmail.com
> Date: Sat, 27 Aug 2022 12:33:49 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> 
> > Cc: 57433@debbugs.gnu.org
> > From: Gerd Möllmann <gerd.moellmann@gmail.com>
> > Date: Sat, 27 Aug 2022 10:53:37 +0200
> > 
> > Gerd Möllmann <gerd.moellmann@gmail.com> writes:
> > 
> > > Gerd Möllmann <gerd.moellmann@gmail.com> writes:
> > >
> > > a71c05b44de74fe16691f680df34c4534992e472 is the first bad commit
> > >
> > > I will look at the changes in simple.el soon if no-one beats me.
> > 
> > Hm, I must admit that I don't understand the change in line-move, or
> > better said the context in which it was made.  So if I fix this I
> > probably break some optimization.
> 
> The changes in line-move should only affect buffers with very long
> lines.

Does Magit turn on truncate-lines in the affected buffer, or cause
them be truncated due to partial-width windows?  If it does, I might
have an idea what could cause the OP's problem.

bug#57433: Emacs no longer moves point into visible port of the buffer

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

> Does Magit turn on truncate-lines in the affected buffer, or cause
> them be truncated due to partial-width windows?  If it does, I might
> have an idea what could cause the OP's problem.

Yes, truncate-lines it t.

bug#57433: Emacs no longer moves point into visible port of the buffer

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: 57433@debbugs.gnu.org,  pogonyshev@gmail.com
> Date: Sat, 27 Aug 2022 12:02:32 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Does Magit turn on truncate-lines in the affected buffer, or cause
> > them be truncated due to partial-width windows?  If it does, I might
> > have an idea what could cause the OP's problem.
> 
> Yes, truncate-lines it t.

In that case, can you see if the patch below solves the original issue
with Magit?

diff --git a/lisp/simple.el b/lisp/simple.el
index ee765c8..d18d54c 100644
--- a/lisp/simple.el
+++ b/lisp/simple.el
@@ -7717,13 +7717,15 @@ line-move
 	       (not goal-column)
                ;; Lines aren't truncated.
                (not
-                (or truncate-lines
-                    (and (integerp truncate-partial-width-windows)
-                         (< (window-width)
-                            truncate-partial-width-windows))
-                    (and truncate-partial-width-windows
-                         (not (integerp truncate-partial-width-windows))
-                         (not (window-full-width-p)))))
+                (and
+                 (or truncate-lines
+                     (and (integerp truncate-partial-width-windows)
+                          (< (window-total-width)
+                             truncate-partial-width-windows))
+                     (and truncate-partial-width-windows
+                          (not (integerp truncate-partial-width-windows))
+                          (not (window-full-width-p))))
+                 (long-line-optimizations-p)))
 	       ;; When the text in the window is scrolled to the left,
 	       ;; display-based motion doesn't make sense (because each
 	       ;; logical line occupies exactly one screen line).

bug#57433: Emacs no longer moves point into visible port of the buffer

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Gerd Möllmann <gerd.moellmann@gmail.com>
>> Cc: 57433@debbugs.gnu.org,  pogonyshev@gmail.com
>> Date: Sat, 27 Aug 2022 12:02:32 +0200
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > Does Magit turn on truncate-lines in the affected buffer, or cause
>> > them be truncated due to partial-width windows?  If it does, I might
>> > have an idea what could cause the OP's problem.
>> 
>> Yes, truncate-lines it t.
>
> In that case, can you see if the patch below solves the original issue
> with Magit?

Works for me.  Thanks!

bug#57433: Emacs no longer moves point into visible port of the buffer

[Eli Zaretskii]

> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> Cc: 57433@debbugs.gnu.org,  pogonyshev@gmail.com
> Date: Sat, 27 Aug 2022 12:43:18 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> >> Cc: 57433@debbugs.gnu.org,  pogonyshev@gmail.com
> >> Date: Sat, 27 Aug 2022 12:02:32 +0200
> >> 
> >> Eli Zaretskii <eliz@gnu.org> writes:
> >> 
> >> > Does Magit turn on truncate-lines in the affected buffer, or cause
> >> > them be truncated due to partial-width windows?  If it does, I might
> >> > have an idea what could cause the OP's problem.
> >> 
> >> Yes, truncate-lines it t.
> >
> > In that case, can you see if the patch below solves the original issue
> > with Magit?
> 
> Works for me.  Thanks!

Thanks, installed.

Paul, please see that your original problem is now solved.

(Btw, I think it's a subtle bug in Magit that it assumes cursor motion
will never end up on invisible text in such buffers.)

bug#57433: Emacs no longer moves point into visible port of the buffer

[Paul Pogonyshev]

[-- Attachment #1: Type: text/plain, Size: 2011 bytes --]

> Paul, please see that your original problem is now solved.

No, don't see any change in behavior.

> (Btw, I think it's a subtle bug in Magit that it assumes cursor motion
> will never end up on invisible text in such buffers.)

I don't think it is a bug. It is explicitly promised in Elisp manual as I
see:

> 22.6 Adjusting Point After Commands
> ===================================
>
> Emacs cannot display the cursor when point is in the middle of a
> sequence of text that has the ‘display’ or ‘composition’ property, or is
> *invisible*.  Therefore, after a command finishes and returns to the
> command loop, if point is within such a sequence, the command loop
> normally moves point to the edge of the sequence, making this sequence
> effectively intangible.

That's why, btw, I couldn't create a testcase: I don't know how to emulate
command loop.

Paul

On Sat, 27 Aug 2022 at 13:08, Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Gerd Möllmann <gerd.moellmann@gmail.com>
> > Cc: 57433@debbugs.gnu.org,  pogonyshev@gmail.com
> > Date: Sat, 27 Aug 2022 12:43:18 +0200
> >
> > Eli Zaretskii <eliz@gnu.org> writes:
> >
> > >> From: Gerd Möllmann <gerd.moellmann@gmail.com>
> > >> Cc: 57433@debbugs.gnu.org,  pogonyshev@gmail.com
> > >> Date: Sat, 27 Aug 2022 12:02:32 +0200
> > >>
> > >> Eli Zaretskii <eliz@gnu.org> writes:
> > >>
> > >> > Does Magit turn on truncate-lines in the affected buffer, or cause
> > >> > them be truncated due to partial-width windows?  If it does, I might
> > >> > have an idea what could cause the OP's problem.
> > >>
> > >> Yes, truncate-lines it t.
> > >
> > > In that case, can you see if the patch below solves the original issue
> > > with Magit?
> >
> > Works for me.  Thanks!
>
> Thanks, installed.
>
> Paul, please see that your original problem is now solved.
>
> (Btw, I think it's a subtle bug in Magit that it assumes cursor motion
> will never end up on invisible text in such buffers.)
>

[-- Attachment #2: Type: text/html, Size: 3232 bytes --]

bug#57433: Emacs no longer moves point into visible port of the buffer

[Gerd Möllmann]

Paul Pogonyshev <pogonyshev@gmail.com> writes:

>> Paul, please see that your original problem is now solved.
>
> No, don't see any change in behavior.

Strange.  Could you please double-check that you have commit
a2d62456a7b8da27fb9b64f71b6ce588f2d73287 in your worktree?

bug#57433: Emacs no longer moves point into visible port of the buffer

[Paul Pogonyshev]

[-- Attachment #1: Type: text/plain, Size: 1312 bytes --]

> Strange.  Could you please double-check that you have commit
> a2d62456a7b8da27fb9b64f71b6ce588f2d73287 in your worktree?

I have rebuilt Emacs and now it works. I previously tried with applying
the patch from "In that case, can you see if the patch below solves the
original issue with Magit?" locally, but apparently did something wrong.

So yes, it works for me now, sorry.

> I think you are reading too much into what that text says.  It says
> that point moves to the edge of such a sequence, and that is what
> happens here.  No more, no less.

Yeah, I see now that if you move with C-b, the point disappears within
the diff for the previous file, even with Emacs 28, and then Magit
misbehaves. To be honest, I'm not sure if that is useful (or maybe there
is some reason for moving "*to the edge*" rather than *outside* an
invisible sequence), but that's certainly not a regression in Emacs.

Paul


On Sat, 27 Aug 2022 at 16:28, Gerd Möllmann <gerd.moellmann@gmail.com>
wrote:

> Paul Pogonyshev <pogonyshev@gmail.com> writes:
>
> >> Paul, please see that your original problem is now solved.
> >
> > No, don't see any change in behavior.
>
> Strange.  Could you please double-check that you have commit
> a2d62456a7b8da27fb9b64f71b6ce588f2d73287 in your worktree?
>

[-- Attachment #2: Type: text/html, Size: 1976 bytes --]

bug#57433: Emacs no longer moves point into visible port of the buffer

[Eli Zaretskii]

> From: Paul Pogonyshev <pogonyshev@gmail.com>
> Date: Sat, 27 Aug 2022 17:47:32 +0200
> Cc: Eli Zaretskii <eliz@gnu.org>, 57433@debbugs.gnu.org
> 
> > Strange.  Could you please double-check that you have commit
> > a2d62456a7b8da27fb9b64f71b6ce588f2d73287 in your worktree?
> 
> I have rebuilt Emacs and now it works. I previously tried with applying
> the patch from "In that case, can you see if the patch below solves the
> original issue with Magit?" locally, but apparently did something wrong.
> 
> So yes, it works for me now, sorry.

No need to be sorry, thanks for testing the fix.  I'm therefore
closing the bug.

> > I think you are reading too much into what that text says.  It says
> > that point moves to the edge of such a sequence, and that is what
> > happens here.  No more, no less.
> 
> Yeah, I see now that if you move with C-b, the point disappears within
> the diff for the previous file, even with Emacs 28, and then Magit
> misbehaves. To be honest, I'm not sure if that is useful (or maybe there
> is some reason for moving "to the edge" rather than outside an
> invisible sequence), but that's certainly not a regression in Emacs.

The "reason" is largely a side-effect of the implementation, I
suspect.

bug#57433: Emacs no longer moves point into visible port of the buffer

[Paul Pogonyshev]

[-- Attachment #1: Type: text/plain, Size: 2733 bytes --]

> The "reason" is largely a side-effect of the implementation, I
> suspect.

Would it then not be a good idea to improve it (not as part of this bug,
of course)? E.g. currently the behavior is like below.

Setup: Magit buffer with staged files, at least two. It looks like this:

Staged changes (N)
modified    file1
[invisible diff of file1]
modified    file2
[invisible diff of file2]

so essentially I, as a user, only see this in UI:

Staged changes (N)
modified    file1
modified    file2

Now, I put the point at the beginning of the third line ([^] is the point):

Staged changes (N)
modified    file1
[^]modified    file2

I press C-b, expecting that the point is moved to the end of the
previous line that I see. But it is not moved there, this is the result:

Staged changes (N)
modified    file1
[^]modified    file2

In reality, the point is in the invisible (to me, as the user) diff for
file1.
If I press C-b again, only now point is moved like I expected it to be
moved the last time:

Staged changes (N)
modified    file1[^]
modified    file2

So, the presence of invisible text between lines 2 and 3 changes the
way C-b behaves and breaks my (as a user of Emacs) expectations,
disrupting established editing flow. I'd say that the invisible text
should count as implementation details and shouldn't affect user-
visible results at all.

What do you think?

Paul


On Sat, 27 Aug 2022 at 17:55, Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Paul Pogonyshev <pogonyshev@gmail.com>
> > Date: Sat, 27 Aug 2022 17:47:32 +0200
> > Cc: Eli Zaretskii <eliz@gnu.org>, 57433@debbugs.gnu.org
> >
> > > Strange.  Could you please double-check that you have commit
> > > a2d62456a7b8da27fb9b64f71b6ce588f2d73287 in your worktree?
> >
> > I have rebuilt Emacs and now it works. I previously tried with applying
> > the patch from "In that case, can you see if the patch below solves the
> > original issue with Magit?" locally, but apparently did something wrong.
> >
> > So yes, it works for me now, sorry.
>
> No need to be sorry, thanks for testing the fix.  I'm therefore
> closing the bug.
>
> > > I think you are reading too much into what that text says.  It says
> > > that point moves to the edge of such a sequence, and that is what
> > > happens here.  No more, no less.
> >
> > Yeah, I see now that if you move with C-b, the point disappears within
> > the diff for the previous file, even with Emacs 28, and then Magit
> > misbehaves. To be honest, I'm not sure if that is useful (or maybe there
> > is some reason for moving "to the edge" rather than outside an
> > invisible sequence), but that's certainly not a regression in Emacs.
>
> The "reason" is largely a side-effect of the implementation, I
> suspect.
>

[-- Attachment #2: Type: text/html, Size: 4091 bytes --]

bug#57433: Emacs no longer moves point into visible port of the buffer

[Paul Pogonyshev]

[-- Attachment #1: Type: text/plain, Size: 3071 bytes --]

Actually, disregard that. It appears to be a Magit bug. If I switch the
buffer
to any other mode, everything behaves as expected.

Paul

On Sat, 27 Aug 2022 at 18:06, Paul Pogonyshev <pogonyshev@gmail.com> wrote:

> > The "reason" is largely a side-effect of the implementation, I
> > suspect.
>
> Would it then not be a good idea to improve it (not as part of this bug,
> of course)? E.g. currently the behavior is like below.
>
> Setup: Magit buffer with staged files, at least two. It looks like this:
>
> Staged changes (N)
> modified    file1
> [invisible diff of file1]
> modified    file2
> [invisible diff of file2]
>
> so essentially I, as a user, only see this in UI:
>
> Staged changes (N)
> modified    file1
> modified    file2
>
> Now, I put the point at the beginning of the third line ([^] is the point):
>
> Staged changes (N)
> modified    file1
> [^]modified    file2
>
> I press C-b, expecting that the point is moved to the end of the
> previous line that I see. But it is not moved there, this is the result:
>
> Staged changes (N)
> modified    file1
> [^]modified    file2
>
> In reality, the point is in the invisible (to me, as the user) diff for
> file1.
> If I press C-b again, only now point is moved like I expected it to be
> moved the last time:
>
> Staged changes (N)
> modified    file1[^]
> modified    file2
>
> So, the presence of invisible text between lines 2 and 3 changes the
> way C-b behaves and breaks my (as a user of Emacs) expectations,
> disrupting established editing flow. I'd say that the invisible text
> should count as implementation details and shouldn't affect user-
> visible results at all.
>
> What do you think?
>
> Paul
>
>
> On Sat, 27 Aug 2022 at 17:55, Eli Zaretskii <eliz@gnu.org> wrote:
>
>> > From: Paul Pogonyshev <pogonyshev@gmail.com>
>> > Date: Sat, 27 Aug 2022 17:47:32 +0200
>> > Cc: Eli Zaretskii <eliz@gnu.org>, 57433@debbugs.gnu.org
>> >
>> > > Strange.  Could you please double-check that you have commit
>> > > a2d62456a7b8da27fb9b64f71b6ce588f2d73287 in your worktree?
>> >
>> > I have rebuilt Emacs and now it works. I previously tried with applying
>> > the patch from "In that case, can you see if the patch below solves the
>> > original issue with Magit?" locally, but apparently did something wrong.
>> >
>> > So yes, it works for me now, sorry.
>>
>> No need to be sorry, thanks for testing the fix.  I'm therefore
>> closing the bug.
>>
>> > > I think you are reading too much into what that text says.  It says
>> > > that point moves to the edge of such a sequence, and that is what
>> > > happens here.  No more, no less.
>> >
>> > Yeah, I see now that if you move with C-b, the point disappears within
>> > the diff for the previous file, even with Emacs 28, and then Magit
>> > misbehaves. To be honest, I'm not sure if that is useful (or maybe there
>> > is some reason for moving "to the edge" rather than outside an
>> > invisible sequence), but that's certainly not a regression in Emacs.
>>
>> The "reason" is largely a side-effect of the implementation, I
>> suspect.
>>
>

[-- Attachment #2: Type: text/html, Size: 4635 bytes --]

bug#57433: Emacs no longer moves point into visible port of the buffer

[Eli Zaretskii]

> From: Paul Pogonyshev <pogonyshev@gmail.com>
> Date: Sat, 27 Aug 2022 18:12:49 +0200
> Cc: Gerd Möllmann <gerd.moellmann@gmail.com>, 
> 	57433-done@debbugs.gnu.org
> 
> Actually, disregard that. It appears to be a Magit bug. If I switch the buffer
> to any other mode, everything behaves as expected.

Right.  I suspect that Magit puts something else there, not just the
invisible property.  What does describe-text-properties say at that
point?

bug#57433: Emacs no longer moves point into visible port of the buffer

[Paul Pogonyshev]

[-- Attachment #1: Type: text/plain, Size: 1690 bytes --]

> Now, I put the point at the beginning of the third line ([^] is the
point):

Text content at position 32867:

There is an overlay here:
 From 32867 to 32901
  evaporate            t
  font-lock-face       magit-diff-file-heading-highlight

There are text properties here:
  font-lock-face       magit-diff-file-heading
  keymap               [Show]
  magit-section        [Show]

> I press C-b, expecting that the point is moved to the end of the
> previous line that I see. But it is not moved there

Text content at position 29881:

There is an overlay here:
 From 29881 to 32867
  evaporate            t
  invisible            t

There are text properties here:
  font-lock-face       magit-diff-hunk-heading
  keymap               [Show]
  magit-section        [Show]

> If I press C-b again, only now point is moved

Text content at position 29880:

There is an overlay here:
 From 29858 to 29881
  evaporate            t
  font-lock-face       magit-diff-file-heading-highlight

There are text properties here:
  keymap               [Show]
  magit-section        [Show]

Paul

On Sat, 27 Aug 2022 at 18:28, Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Paul Pogonyshev <pogonyshev@gmail.com>
> > Date: Sat, 27 Aug 2022 18:12:49 +0200
> > Cc: Gerd Möllmann <gerd.moellmann@gmail.com>,
> >       57433-done@debbugs.gnu.org
> >
> > Actually, disregard that. It appears to be a Magit bug. If I switch the
> buffer
> > to any other mode, everything behaves as expected.
>
> Right.  I suspect that Magit puts something else there, not just the
> invisible property.  What does describe-text-properties say at that
> point?
>

[-- Attachment #2: Type: text/html, Size: 2505 bytes --]

bug#57433: Emacs no longer moves point into visible port of the buffer

[Paul Pogonyshev]

[-- Attachment #1: Type: text/plain, Size: 1963 bytes --]

Also, the value of `magit-section' is different all three times, if that's
important.

Paul

On Sat, 27 Aug 2022 at 18:32, Paul Pogonyshev <pogonyshev@gmail.com> wrote:

> > Now, I put the point at the beginning of the third line ([^] is the
> point):
>
> Text content at position 32867:
>
> There is an overlay here:
>  From 32867 to 32901
>   evaporate            t
>   font-lock-face       magit-diff-file-heading-highlight
>
> There are text properties here:
>   font-lock-face       magit-diff-file-heading
>   keymap               [Show]
>   magit-section        [Show]
>
> > I press C-b, expecting that the point is moved to the end of the
> > previous line that I see. But it is not moved there
>
> Text content at position 29881:
>
> There is an overlay here:
>  From 29881 to 32867
>   evaporate            t
>   invisible            t
>
> There are text properties here:
>   font-lock-face       magit-diff-hunk-heading
>   keymap               [Show]
>   magit-section        [Show]
>
> > If I press C-b again, only now point is moved
>
> Text content at position 29880:
>
> There is an overlay here:
>  From 29858 to 29881
>   evaporate            t
>   font-lock-face       magit-diff-file-heading-highlight
>
> There are text properties here:
>   keymap               [Show]
>   magit-section        [Show]
>
> Paul
>
> On Sat, 27 Aug 2022 at 18:28, Eli Zaretskii <eliz@gnu.org> wrote:
>
>> > From: Paul Pogonyshev <pogonyshev@gmail.com>
>> > Date: Sat, 27 Aug 2022 18:12:49 +0200
>> > Cc: Gerd Möllmann <gerd.moellmann@gmail.com>,
>> >       57433-done@debbugs.gnu.org
>> >
>> > Actually, disregard that. It appears to be a Magit bug. If I switch the
>> buffer
>> > to any other mode, everything behaves as expected.
>>
>> Right.  I suspect that Magit puts something else there, not just the
>> invisible property.  What does describe-text-properties say at that
>> point?
>>
>

[-- Attachment #2: Type: text/html, Size: 3003 bytes --]

bug#57433: Emacs no longer moves point into visible port of the buffer

[Eli Zaretskii]

> From: Paul Pogonyshev <pogonyshev@gmail.com>
> Date: Sat, 27 Aug 2022 18:32:15 +0200
> Cc: Gerd Möllmann <gerd.moellmann@gmail.com>, 
> 	57433@debbugs.gnu.org
> 
> > Now, I put the point at the beginning of the third line ([^] is the point):
> 
> Text content at position 32867:
> 
> There is an overlay here:
>  From 32867 to 32901
>   evaporate            t
>   font-lock-face       magit-diff-file-heading-highlight
> 
> There are text properties here:
>   font-lock-face       magit-diff-file-heading
>   keymap               [Show]
>   magit-section        [Show]

Maybe the 'keymap' part?  Could it be that C-b has a binding in that
keymap?

bug#57433: Emacs no longer moves point into visible port of the buffer

[Paul Pogonyshev]

[-- Attachment #1: Type: text/plain, Size: 2841 bytes --]

There is no mode binding (`C-h k C-b' returns `backward-char'). I don't know
how to read this (result of showing the keymap in `describe-text-properties'
buffer; at step 1:

(keymap keymap
        (3 keymap
           (5 . magit-diff-edit-hunk-commit)
           (20 . magit-diff-trace-definition))
        (67 . magit-commit-add-log)
        (38 . magit-do-async-shell-command)
        (117 . magit-unstage)
        (115 . magit-stage)
        (97 . magit-apply)
        (remap keymap
               (magit-revert-no-commit . magit-reverse)
               (magit-delete-thing . magit-discard)
               (magit-visit-thing . magit-diff-visit-file))
        (24 keymap
            (97 . magit-add-change-log-entry)
            (53 keymap
                (return . magit-diff-visit-file-other-frame))
            (52 keymap
                (97 . magit-add-change-log-entry-other-window)
                (return . magit-diff-visit-file-other-window)))
        (C-return . magit-diff-visit-worktree-file)
        (10 . magit-diff-visit-worktree-file))

At step 2:

(keymap
 (3 keymap
    (94 keymap
        (108 . magit-smerge-keep-lower)
        (98 . magit-smerge-keep-base)
        (117 . magit-smerge-keep-upper)
        (13 . magit-smerge-keep-current)))
 keymap
 (3 keymap
    (5 . magit-diff-edit-hunk-commit)
    (20 . magit-diff-trace-definition))
 (67 . magit-commit-add-log)
 (38 . magit-do-async-shell-command)
 (117 . magit-unstage)
 (115 . magit-stage)
 (97 . magit-apply)
 (remap keymap
        (magit-revert-no-commit . magit-reverse)
        (magit-delete-thing . magit-discard)
        (magit-visit-thing . magit-diff-visit-file))
 (24 keymap
     (97 . magit-add-change-log-entry)
     (53 keymap
         (return . magit-diff-visit-file-other-frame))
     (52 keymap
         (97 . magit-add-change-log-entry-other-window)
         (return . magit-diff-visit-file-other-window)))
 (C-return . magit-diff-visit-worktree-file)

Paul
 (10 . magit-diff-visit-worktree-file))

On Sat, 27 Aug 2022 at 18:56, Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Paul Pogonyshev <pogonyshev@gmail.com>
> > Date: Sat, 27 Aug 2022 18:32:15 +0200
> > Cc: Gerd Möllmann <gerd.moellmann@gmail.com>,
> >       57433@debbugs.gnu.org
> >
> > > Now, I put the point at the beginning of the third line ([^] is the
> point):
> >
> > Text content at position 32867:
> >
> > There is an overlay here:
> >  From 32867 to 32901
> >   evaporate            t
> >   font-lock-face       magit-diff-file-heading-highlight
> >
> > There are text properties here:
> >   font-lock-face       magit-diff-file-heading
> >   keymap               [Show]
> >   magit-section        [Show]
>
> Maybe the 'keymap' part?  Could it be that C-b has a binding in that
> keymap?
>

[-- Attachment #2: Type: text/html, Size: 3865 bytes --]

bug#57433: Emacs no longer moves point into visible port of the buffer

[Paul Pogonyshev]

[-- Attachment #1: Type: text/plain, Size: 3820 bytes --]

By the way, here is a simple way to test Magit without installing it in your
Emacs. However, you will need Eldev (external relative to Emacs).

1. Install Eldev (https://github.com/doublep/eldev#installation)

2. Check out Magit sources, e.g. to `~/git/magit'

3. Run `$ cd ~/git/magit/*lisp*' (I don't know why they have this layout)

4. Create file `Eldev-local' with the following contents (since the
project doesn't use Eldev, you have to do this manually):

(setf eldev-project-main-file "magit.el")
(eldev-use-package-archive 'gnu-elpa)
(eldev-use-package-archive 'melpa)

5. Run e.g. like this: `$ eldev emacs --eval "(magit-status-setup-buffer
\"~/emacs\")" ',
assuming `~/emacs' is a Git checkout you work on.

Paul

On Sat, 27 Aug 2022 at 19:01, Paul Pogonyshev <pogonyshev@gmail.com> wrote:

> There is no mode binding (`C-h k C-b' returns `backward-char'). I don't
> know
> how to read this (result of showing the keymap in
> `describe-text-properties'
> buffer; at step 1:
>
> (keymap keymap
>         (3 keymap
>            (5 . magit-diff-edit-hunk-commit)
>            (20 . magit-diff-trace-definition))
>         (67 . magit-commit-add-log)
>         (38 . magit-do-async-shell-command)
>         (117 . magit-unstage)
>         (115 . magit-stage)
>         (97 . magit-apply)
>         (remap keymap
>                (magit-revert-no-commit . magit-reverse)
>                (magit-delete-thing . magit-discard)
>                (magit-visit-thing . magit-diff-visit-file))
>         (24 keymap
>             (97 . magit-add-change-log-entry)
>             (53 keymap
>                 (return . magit-diff-visit-file-other-frame))
>             (52 keymap
>                 (97 . magit-add-change-log-entry-other-window)
>                 (return . magit-diff-visit-file-other-window)))
>         (C-return . magit-diff-visit-worktree-file)
>         (10 . magit-diff-visit-worktree-file))
>
> At step 2:
>
> (keymap
>  (3 keymap
>     (94 keymap
>         (108 . magit-smerge-keep-lower)
>         (98 . magit-smerge-keep-base)
>         (117 . magit-smerge-keep-upper)
>         (13 . magit-smerge-keep-current)))
>  keymap
>  (3 keymap
>     (5 . magit-diff-edit-hunk-commit)
>     (20 . magit-diff-trace-definition))
>  (67 . magit-commit-add-log)
>  (38 . magit-do-async-shell-command)
>  (117 . magit-unstage)
>  (115 . magit-stage)
>  (97 . magit-apply)
>  (remap keymap
>         (magit-revert-no-commit . magit-reverse)
>         (magit-delete-thing . magit-discard)
>         (magit-visit-thing . magit-diff-visit-file))
>  (24 keymap
>      (97 . magit-add-change-log-entry)
>      (53 keymap
>          (return . magit-diff-visit-file-other-frame))
>      (52 keymap
>          (97 . magit-add-change-log-entry-other-window)
>          (return . magit-diff-visit-file-other-window)))
>  (C-return . magit-diff-visit-worktree-file)
>
> Paul
>  (10 . magit-diff-visit-worktree-file))
>
> On Sat, 27 Aug 2022 at 18:56, Eli Zaretskii <eliz@gnu.org> wrote:
>
>> > From: Paul Pogonyshev <pogonyshev@gmail.com>
>> > Date: Sat, 27 Aug 2022 18:32:15 +0200
>> > Cc: Gerd Möllmann <gerd.moellmann@gmail.com>,
>> >       57433@debbugs.gnu.org
>> >
>> > > Now, I put the point at the beginning of the third line ([^] is the
>> point):
>> >
>> > Text content at position 32867:
>> >
>> > There is an overlay here:
>> >  From 32867 to 32901
>> >   evaporate            t
>> >   font-lock-face       magit-diff-file-heading-highlight
>> >
>> > There are text properties here:
>> >   font-lock-face       magit-diff-file-heading
>> >   keymap               [Show]
>> >   magit-section        [Show]
>>
>> Maybe the 'keymap' part?  Could it be that C-b has a binding in that
>> keymap?
>>
>

[-- Attachment #2: Type: text/html, Size: 5304 bytes --]

bug#57433: Emacs no longer moves point into visible port of the buffer

[Eli Zaretskii]

> From: Paul Pogonyshev <pogonyshev@gmail.com>
> Date: Sat, 27 Aug 2022 18:06:59 +0200
> Cc: Gerd Möllmann <gerd.moellmann@gmail.com>, 
> 	57433-done@debbugs.gnu.org
> 
> Setup: Magit buffer with staged files, at least two. It looks like this:
> 
> Staged changes (N)
> modified    file1
> [invisible diff of file1]
> modified    file2
> [invisible diff of file2]
> 
> so essentially I, as a user, only see this in UI:
> 
> Staged changes (N)
> modified    file1
> modified    file2
> 
> Now, I put the point at the beginning of the third line ([^] is the point):
> 
> Staged changes (N)
> modified    file1
> [^]modified    file2
> 
> I press C-b, expecting that the point is moved to the end of the
> previous line that I see. But it is not moved there, this is the result:
> 
> Staged changes (N)
> modified    file1
> [^]modified    file2
> 
> In reality, the point is in the invisible (to me, as the user) diff for file1.
> If I press C-b again, only now point is moved like I expected it to be
> moved the last time:
> 
> Staged changes (N)
> modified    file1[^]
> modified    file2

I cannot reproduce this, if I try to simulate it by having those 4
lines with two lines invisible.  So I think there's something else at
work here.  If you can show a simple recipe that doesn't involve
Magit, I can look into what happens there.  (And it probably warrants
a separate bug report.)

What I did was to have this text:

  modified    file1
  [invisible diff of file1]
  modified    file2
  [invisible diff of file2]

(without the indentation) in a buffer, and then I manually put the
invisible text property on the 2nd and the 4th lines.  Then I moved
the cursor with C-b from the beginning of "modified file2", and the
cursor was displayed at the end of the previous visible line, the one
which says "modified file1".

> So, the presence of invisible text between lines 2 and 3 changes the
> way C-b behaves and breaks my (as a user of Emacs) expectations,
> disrupting established editing flow. I'd say that the invisible text
> should count as implementation details and shouldn't affect user-
> visible results at all.
> 
> What do you think?

It works for me the way you expect it to work, so I'm not yet sure
what's going on in the Magit buffer.

bug#57433: Emacs no longer moves point into visible port of the buffer

[Eli Zaretskii]

> From: Paul Pogonyshev <pogonyshev@gmail.com>
> Date: Sat, 27 Aug 2022 16:07:49 +0200
> Cc: Gerd Möllmann <gerd.moellmann@gmail.com>, 
> 	57433@debbugs.gnu.org
> 
> > Paul, please see that your original problem is now solved.
> 
> No, don't see any change in behavior.

Then I guess what you see is a separate and different problem.

> > (Btw, I think it's a subtle bug in Magit that it assumes cursor motion
> > will never end up on invisible text in such buffers.)
> 
> I don't think it is a bug. It is explicitly promised in Elisp manual as I see:
> 
> > 22.6 Adjusting Point After Commands
> > ===================================
> > 
> > Emacs cannot display the cursor when point is in the middle of a
> > sequence of text that has the ‘display’ or ‘composition’ property, or is
> > invisible.  Therefore, after a command finishes and returns to the
> > command loop, if point is within such a sequence, the command loop
> > normally moves point to the edge of the sequence, making this sequence
> > effectively intangible.

I think you are reading too much into what that text says.  It says
that point moves to the edge of such a sequence, and that is what
happens here.  No more, no less.

> That's why, btw, I couldn't create a testcase: I don't know how to emulate
> command loop.

Exactly.

bug#57433: Emacs no longer moves point into visible port of the buffer

[Eli Zaretskii]

> From: Paul Pogonyshev <pogonyshev@gmail.com>
> Date: Fri, 26 Aug 2022 18:32:14 +0200
> 
> Git commit 233c91d32a. Has been broken at least for a few days: my previous build also exhibited this.
> 
> No idea how to create a short test in Elisp.
> 
> From user point of view: open a Magit buffer in a directory with at least two locally modified and uncommitted
> files. The buffer shows sth. like this:
> 
> ...
> Staged changes (N)
> modified   file1
> modified   file2
> ...
> 
> Move the point with C-n to the line with `file2'. Press TAB. Magit section for the file is not opened. Use `M-x
> describe-char'. From the resulting buffer you can say that the point is actually on an invisible char '@', i.e.
> start of (hidden) diff for the first file.

Can you tell what is the actual buffer text and where are the
invisible text properties on it?