Odd behavior when moving point over invisible text

Odd behavior when moving point over invisible text

[Mats Lidell]

Hi,

When writing a unit test I wanted to move point over invisible (hidden) text and
got into this strange behavior:

1. Insert these two lines in an elisp file:
    (defun func ()
      (point)) 
2. Set outline-minor-mode { M-x outline-minor-mode }
3. Goto first line and do { M-x outline-hide-subtree }
4. Goto first . in the ellipsis after the parentheses () and try either to use
   the keyboard with C-f or use the command (forward-char) to move over the
   ellipsis.

With C-f point is moved over the ellipsis but with forward-char point remains
put!? I suspect it is due to point-adjustment but I was expecting both of these
to produce the same result, i.e. move over the ellipsis. Not to mention that
forward-char is also bound to C-f!?

I tried this with 27.2, 28.2 and current master with the same result.

%% Mats

Re: Odd behavior when moving point over invisible text

[Karl Fogel]

On 04 Jun 2023, Mats Lidell wrote:
>When writing a unit test I wanted to move point over invisible 
>(hidden) text and
>got into this strange behavior:
>
>1. Insert these two lines in an elisp file:
>    (defun func ()
>      (point)) 
>2. Set outline-minor-mode { M-x outline-minor-mode }
>3. Goto first line and do { M-x outline-hide-subtree }
>4. Goto first . in the ellipsis after the parentheses () and try 
>either to use
>   the keyboard with C-f or use the command (forward-char) to 
>   move over the
>   ellipsis.
>
>With C-f point is moved over the ellipsis but with forward-char 
>point remains
>put!? I suspect it is due to point-adjustment but I was expecting 
>both of these
>to produce the same result, i.e. move over the ellipsis. Not to 
>mention that
>forward-char is also bound to C-f!?
>
>I tried this with 27.2, 28.2 and current master with the same 
>result.

This reproduces for me with current master 
(`emacs-repository-version' is 
"f947a0219bb6e43966e0e4e61ad6a15b0ed13e18").

The reason I'm looking into your report was because I thought it 
might be related to a known bug in indent.c:check_display_width() 
that affects scan_for_column() and results in a similar "point 
doesn't move forward by the right amount" misbehavior [1].  I 
don't yet know if the problem you're describing has the same 
underlying cause as [1]; I'd have to trace in and I don't know 
when I'll get time to do that.  But I thought I'd reply here in 
case they are related; maybe that would save you some time.

I looked around in the code a bit for your bug. 
cmds.c:Fforward_char() just calls move_point(), which in turn 
looks like it boils down to calling intervals.c:set_point(), which 
gets us to set_point_both() and that appears to finally be where 
some interesting action starts.  So is there a call to 
scan_for_column(), or something else that gets us to 
check_display_width(), somewhere within set_point_both() in the 
call stack?  Maybe?  I don't yet know.

I also don't know why `forward-char' is behaving differently in 
your reproduction recipe when called via `C-f' versus other ways. 
That's quite odd IMHO!  Both `M-x forward-char' and direct 
minibuffer evaluation of `(forward-char 1)' result in point 
staying on the first dot, just as you said, which is position 15 
in the buffer.  Whereas typing `C-f' moves point over the whole 
ellipsis and goes to position 28 in the buffer.  What's up with 
that?  There doesn't seem to be any advice added to anything by 
outline.el (disclaimer: I'm not very familiar with the advice 
mechanism in Emacs).  For example, calling `(ad-has-any-advice 
'forward-char)' within the reproduction buffer returns nil.

Best regards,
-Karl

[1] 
https://lists.gnu.org/archive/html/emacs-devel/2022-11/msg01170.html 
    From: Karl Fogel <kfogel@red-bean.com>
    To: Emacs Devel <emacs-devel@gnu.org>
    Subject: An interesting line-motion bug.
    Date: Wed, 16 Nov 2022 23:38:32 -0600
    Message-ID: <87cz9mywbr.fsf@red-bean.com>

Re: Odd behavior when moving point over invisible text

[Eli Zaretskii]

> From: Mats Lidell <matsl@gnu.org>
> Date: Sun, 04 Jun 2023 23:43:39 +0200
> 
> 1. Insert these two lines in an elisp file:
>     (defun func ()
>       (point)) 
> 2. Set outline-minor-mode { M-x outline-minor-mode }
> 3. Goto first line and do { M-x outline-hide-subtree }
> 4. Goto first . in the ellipsis after the parentheses () and try either to use
>    the keyboard with C-f or use the command (forward-char) to move over the
>    ellipsis.
> 
> With C-f point is moved over the ellipsis but with forward-char point remains
> put!? I suspect it is due to point-adjustment but I was expecting both of these
> to produce the same result, i.e. move over the ellipsis. Not to mention that
> forward-char is also bound to C-f!?

It's a bug.  Should be fixed now on the master branch.

Re: Odd behavior when moving point over invisible text

[Eli Zaretskii]

> From: Karl Fogel <kfogel@red-bean.com>
> Cc: emacs-devel@gnu.org
> Date: Mon, 05 Jun 2023 02:09:53 -0500
> 
> The reason I'm looking into your report was because I thought it 
> might be related to a known bug in indent.c:check_display_width() 

It isn't.

> I looked around in the code a bit for your bug. 
> cmds.c:Fforward_char() just calls move_point(), which in turn 
> looks like it boils down to calling intervals.c:set_point(), which 
> gets us to set_point_both() and that appears to finally be where 
> some interesting action starts.  So is there a call to 
> scan_for_column(), or something else that gets us to 
> check_display_width(), somewhere within set_point_both() in the 
> call stack?  Maybe?  I don't yet know.

No, those are not relevant.  What happens here is the feature known as
"point-adjustment" kicks in, and attempts to move point out of
invisible text.  See disable-point-adjustment for more details.

> I also don't know why `forward-char' is behaving differently in 
> your reproduction recipe when called via `C-f' versus other ways. 
> That's quite odd IMHO!  Both `M-x forward-char' and direct 
> minibuffer evaluation of `(forward-char 1)' result in point 
> staying on the first dot, just as you said, which is position 15 
> in the buffer.  Whereas typing `C-f' moves point over the whole 
> ellipsis and goes to position 28 in the buffer.  What's up with 
> that?  There doesn't seem to be any advice added to anything by 
> outline.el (disclaimer: I'm not very familiar with the advice 
> mechanism in Emacs).  For example, calling `(ad-has-any-advice 
> 'forward-char)' within the reproduction buffer returns nil.

It's a bug.  Bugs are often odd.  The point-adjustment feature needs
to record the previous position of point, to know in which direction
to move it in order to exit the invisible area: if point moved to a
larger buffer position and entered invisible text, point-adjustment
wants to move it further forward; if it moved towards lower position,
point-adjustment wants to move it back.  The bug was that last point
position was recorded in a global variable, and M-x enters
recursive-edit, which clobbered that variable's value with the value
from the minibuffer.

Re: Odd behavior when moving point over invisible text

[Karl Fogel]

On 05 Jun 2023, Eli Zaretskii wrote:
>The point-adjustment feature needs to record the previous
>position of point, to know in which direction to move it in
>order to exit the invisible area: if point moved to a larger
>buffer position and entered invisible text, point-adjustment
>wants to move it further forward; if it moved towards lower
>position, point-adjustment wants to move it back.  The bug was
>that last point position was recorded in a global variable, and
>M-x enters recursive-edit, which clobbered that variable's value
>with the value from the minibuffer.

Thank you for the clear explanation, Eli.  That all makes sense, 
as does your fix in commit 9a28600a9735.

Best regards,
-Karl

Re: Odd behavior when moving point over invisible text

[Mats Lidell]

> Eli Zaretskii writes:
> The bug was that last point position was recorded in a global variable, and
> M-x enters recursive-edit, which clobbered that variable's value with the
> value from the minibuffer.

Thanks for the clear description and fix.

%% Mats

Re: Odd behavior when moving point over invisible text

[Mats Lidell]

> Eli Zaretskii writes:
> It's a bug.  Should be fixed now on the master branch.

I had a look. Now the behavior looks consistent with using the key binding.

But... There is still a thing that I find a little odd but might be the
desired behavior.

What happens is: If point is before the first dot in the ellipsis and I call
(forward-char) multiple times, from a function or test case, not manually, the
point stays on the first dot (It seems point is really moving in the hidden
text because if the number of times it is repeated is larger than the hidden
text the point comes out on the other side so to say.)

This can also be demonstrated with trying to pass the ellipsis using a prefix
arg to C-f. It is easiest to demonstrate by having a big text portion that is
hidden. Put the point before the ellipsis on the right parenthesis in the
example file and type C-u 10 C-f and notice that point will stop at the first
dot. Put back the point on the parenthesis and type C-f C-f and point moves
over the ellipsis.

%% Mats

Re: Odd behavior when moving point over invisible text

[Eli Zaretskii]

> From: Mats Lidell <matsl@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Tue, 06 Jun 2023 01:30:44 +0200
> 
> But... There is still a thing that I find a little odd but might be the
> desired behavior.
> 
> What happens is: If point is before the first dot in the ellipsis and I call
> (forward-char) multiple times, from a function or test case, not manually, the
> point stays on the first dot (It seems point is really moving in the hidden
> text because if the number of times it is repeated is larger than the hidden
> text the point comes out on the other side so to say.)
> 
> This can also be demonstrated with trying to pass the ellipsis using a prefix
> arg to C-f. It is easiest to demonstrate by having a big text portion that is
> hidden. Put the point before the ellipsis on the right parenthesis in the
> example file and type C-u 10 C-f and notice that point will stop at the first
> dot. Put back the point on the parenthesis and type C-f C-f and point moves
> over the ellipsis.

I don't think I understand what you find odd here.  What exactly are
your expectations from this heuristic?  The commands, like
forward-char, aren't changed, they still move by the exact number of
buffer positions they are told to move.  The heuristic kicks in
_after_ the command exits, and it (the heuristic) tries to figure out
on which end of the invisible region to put point.  When point moves
from "afar" into the invisible region, how to know whether to put it
at the nearest end or the farthest one?

Re: Odd behavior when moving point over invisible text

[Mats Lidell]

> Eli Zaretskii writes:
> I don't think I understand what you find odd here.

My initial "odd feeling" comes from this.

   Put the point one char away from the ellipsis. Press C-f twice and point
   moves up to the ellipsis and over it. If I instead do C-u 2 C-f the point
   will stop at the ellipsis.

> What exactly are your expectations from this heuristic?

I would expect both ways to move the point to produce the same result.

> The commands, like forward-char, aren't changed, they still move by the
> exact number of buffer positions they are told to move.  The heuristic kicks
> in _after_ the command exits, and it (the heuristic) tries to figure out on
> which end of the invisible region to put point.  When point moves from
> "afar" into the invisible region, how to know whether to put it at the
> nearest end or the farthest one?

I understand the difficulties and also the problems related to "fixing" this
is dawning on me. Thanks for the feedback.

For my initial quest to write some unit tests around hidden text I feel better
equipped now. I understand why (forward-char) twice can't be used in the unit
test to move past the ellipsis since that behavior is connected to the
heuristics that will not kick in.

%% Mats

Re: Odd behavior when moving point over invisible text

[Eli Zaretskii]

> From: Mats Lidell <matsl@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Tue, 06 Jun 2023 14:07:07 +0200
> 
>    Put the point one char away from the ellipsis. Press C-f twice and point
>    moves up to the ellipsis and over it. If I instead do C-u 2 C-f the point
>    will stop at the ellipsis.
> 
> > What exactly are your expectations from this heuristic?
> 
> I would expect both ways to move the point to produce the same result.

Why?  In the first case, Emacs knows you stopped between the two
moves, in the second one you didn't.

> For my initial quest to write some unit tests around hidden text I feel better
> equipped now. I understand why (forward-char) twice can't be used in the unit
> test to move past the ellipsis since that behavior is connected to the
> heuristics that will not kick in.

If you are writing tests, you are better off disabling point
adjustments altogether.  See global-disable-point-adjustment.

Re: Odd behavior when moving point over invisible text

[Mats Lidell]

> Eli Zaretskii writes:
> Why?  In the first case, Emacs knows you stopped between the two
> moves, in the second one you didn't.

A user might not be aware of the heuristics in place, view C-u as a repeat
count and be surprised by the outcome.

> If you are writing tests, you are better off disabling point adjustments
> altogether.  See global-disable-point-adjustment.

I understand. I will try to refactor how to write those tests.

%% Mats