bug#9034: point adjustment after `end-of-visual-line' moves to the next line

bug#9034: point adjustment after `end-of-visual-line' moves to the next line

[Dmitry Kurochkin]

Package: emacs
Version: 24.0.50

If a line ends with invisible text, point adjustment moves to the next
line after `end-of-visual-line'.  To reproduce run:

  (progn (switch-to-buffer "test")
         (insert "aline1\nbline2\ncline3\n")
         (put-text-property 7 14 'invisible t) ; region [\nbline2]
         (goto-char (point-min)))

After that in the test buffer do M-: (progn (end-of-visual-line)
(point)).  `end-of-visual-line' moves point to position 14 as expected.
But after that the point is adjusted to position 15, as can be checked
by C-x =.

Sometimes the point is not adjusted to position 15 the first time you
run `end-of-visual-line'.  But it is if you run it more than once.
Perhaps it is related to the last point position.

It is definitely happens because of point adjustment, setting
global-disable-point-adjustment to non-nil fixes the problem.

Also, `move-end-of-line` always behaves correctly, so it it something
specific to `end-of-visual-line' or `vertical-motion'.

The issue was originally reported on the emacs-devel ML [1].

Regards,
  Dmitry

[1] http://lists.gnu.org/archive/html/emacs-devel/2011-07/msg00255.html

bug#9034: point adjustment after `end-of-visual-line' moves to the next line

[Stefan Monnier]

> If a line ends with invisible text, point adjustment moves to the next
> line after `end-of-visual-line'.  To reproduce run:

>   (progn (switch-to-buffer "test")
>          (insert "aline1\nbline2\ncline3\n")
>          (put-text-property 7 14 'invisible t) ; region [\nbline2]
>          (goto-char (point-min)))

> After that in the test buffer do M-: (progn (end-of-visual-line)
> (point)).  `end-of-visual-line' moves point to position 14 as expected.
> But after that the point is adjusted to position 15, as can be checked
> by C-x =.

> Sometimes the point is not adjusted to position 15 the first time you
> run `end-of-visual-line'.  But it is if you run it more than once.
> Perhaps it is related to the last point position.

I can reproduce it with M-: (progn (goto-char 14) > (point))

> Also, `move-end-of-line` always behaves correctly, so it it something
> specific to `end-of-visual-line' or `vertical-motion'.

Apparently not since I can reproduce the problem with just goto-char.
Maybe it has to do with the use of M-: instead.


        Stefan

bug#9034: point adjustment after `end-of-visual-line' moves to the next line

[Dmitry Kurochkin]

On Mon, 11 Jul 2011 00:16:31 -0400, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> > If a line ends with invisible text, point adjustment moves to the next
> > line after `end-of-visual-line'.  To reproduce run:
> 
> >   (progn (switch-to-buffer "test")
> >          (insert "aline1\nbline2\ncline3\n")
> >          (put-text-property 7 14 'invisible t) ; region [\nbline2]
> >          (goto-char (point-min)))
> 
> > After that in the test buffer do M-: (progn (end-of-visual-line)
> > (point)).  `end-of-visual-line' moves point to position 14 as expected.
> > But after that the point is adjusted to position 15, as can be checked
> > by C-x =.
> 
> > Sometimes the point is not adjusted to position 15 the first time you
> > run `end-of-visual-line'.  But it is if you run it more than once.
> > Perhaps it is related to the last point position.
> 
> I can reproduce it with M-: (progn (goto-char 14) > (point))
> 

indeed

> > Also, `move-end-of-line` always behaves correctly, so it it something
> > specific to `end-of-visual-line' or `vertical-motion'.
> 
> Apparently not since I can reproduce the problem with just goto-char.

If visual-line-mode is enabled (so that C-e is mapped to
`end-of-visual-line' instead of `move-end-of-line'),
`end-of-visual-line' works correctly when run with C-e.

> Maybe it has to do with the use of M-: instead.
> 

Seems plausible.

Regards,
  Dmitry

> 
>         Stefan

bug#9034: point adjustment (with invisible text) moves to the next line

[Lars Ingebrigtsen]

Dmitry Kurochkin <dmitry.kurochkin@gmail.com> writes:

>   (progn (switch-to-buffer "test")
>          (insert "aline1\nbline2\ncline3\n")
>          (put-text-property 7 14 'invisible t) ; region [\nbline2]
>          (goto-char (point-min)))
>
> After that in the test buffer do M-: (progn (end-of-visual-line)
> (point)).  `end-of-visual-line' moves point to position 14 as expected.
> But after that the point is adjusted to position 15, as can be checked
> by C-x =.

(I'm going through old bug reports that unfortunately got weren't
resolved at the time.)

This problem is still present in Emacs 28, and as previously noted, any
command you do (with `M-:') makes point advance to the next line.  (It's
not 100% reproducible -- sometimes nothing happens, and sometimes it
advances.)

So `C-e' and then, for instance `M-: 4 RET' will advance point here.

Also as previously noted, if Vglobal_disable_point_adjustment is on,
then this doesn't happen, so the problem almost has to be here:

command_loop_1
...
      if (current_buffer == prev_buffer
	  && XBUFFER (XWINDOW (selected_window)->contents) == current_buffer
	  && last_point_position != PT
	  && NILP (Vdisable_point_adjustment)
	  && NILP (Vglobal_disable_point_adjustment))
	{
	  if (last_point_position > BEGV
	      && last_point_position < ZV
	      && (composition_adjust_point (last_point_position,
					    last_point_position)
		  != last_point_position))
	    /* The last point was temporarily set within a grapheme
	       cluster to prevent automatic composition.  To recover
	       the automatic composition, we must update the
	       display.  */
	    windows_or_buffers_changed = 21;
	  if (!already_adjusted)
	    adjust_point_for_property (last_point_position,
				       MODIFF != prev_modiff);
	}

I haven't tried to debug any further -- is it obvious to anybody what's
failing here, by any chance?  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#9034: point adjustment (with invisible text) moves to the next line

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Wed, 25 Aug 2021 18:32:15 +0200
> Cc: 9034@debbugs.gnu.org
> 
> Also as previously noted, if Vglobal_disable_point_adjustment is on,
> then this doesn't happen, so the problem almost has to be here:
> 
> command_loop_1
> ...
>       if (current_buffer == prev_buffer
> 	  && XBUFFER (XWINDOW (selected_window)->contents) == current_buffer
> 	  && last_point_position != PT
> 	  && NILP (Vdisable_point_adjustment)
> 	  && NILP (Vglobal_disable_point_adjustment))
> 	{
> 	  if (last_point_position > BEGV
> 	      && last_point_position < ZV
> 	      && (composition_adjust_point (last_point_position,
> 					    last_point_position)
> 		  != last_point_position))
> 	    /* The last point was temporarily set within a grapheme
> 	       cluster to prevent automatic composition.  To recover
> 	       the automatic composition, we must update the
> 	       display.  */
> 	    windows_or_buffers_changed = 21;
> 	  if (!already_adjusted)
> 	    adjust_point_for_property (last_point_position,
> 				       MODIFF != prev_modiff);
> 	}
> 
> I haven't tried to debug any further -- is it obvious to anybody what's
> failing here, by any chance?  :-)

This code is chock-full of questionable heuristics, and in particular
tries to guess in which direction to move point out of the invisible
text.  The results are sometimes surprising.  The upside is that it
"mostly works".  But if someone can come up with a better heuristics,
I'm all ears.

bug#9034: point adjustment (with invisible text) moves to the next line

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> This code is chock-full of questionable heuristics, and in particular
> tries to guess in which direction to move point out of the invisible
> text.  The results are sometimes surprising.  The upside is that it
> "mostly works".  But if someone can come up with a better heuristics,
> I'm all ears.

Ah, right.  But...  the super-confusing thing about this is that we're
not moving point in the buffer at all.  Just doing `M-: 4 RET' will
result in this code adjusting point in the buffer.  Doing, for instance,
`M-x foo RET' (where `foo' does nothing) does not result in this point
movement.

So is `M-:' doing some weird thing in the original buffer to trigger
this oddity?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#9034: point adjustment (with invisible text) moves to the next line

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: dmitry.kurochkin@gmail.com,  9034@debbugs.gnu.org
> Date: Thu, 26 Aug 2021 15:48:32 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > This code is chock-full of questionable heuristics, and in particular
> > tries to guess in which direction to move point out of the invisible
> > text.  The results are sometimes surprising.  The upside is that it
> > "mostly works".  But if someone can come up with a better heuristics,
> > I'm all ears.
> 
> Ah, right.  But...  the super-confusing thing about this is that we're
> not moving point in the buffer at all.

The original recipe moved from point-min to an end of a line.

> Just doing `M-: 4 RET' will result in this code adjusting point in
> the buffer.  Doing, for instance, `M-x foo RET' (where `foo' does
> nothing) does not result in this point movement.
> 
> So is `M-:' doing some weird thing in the original buffer to trigger
> this oddity?

Doing M-: in what part of the recipe?

bug#9034: point adjustment (with invisible text) moves to the next line

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> Just doing `M-: 4 RET' will result in this code adjusting point in
>> the buffer.  Doing, for instance, `M-x foo RET' (where `foo' does
>> nothing) does not result in this point movement.
>> 
>> So is `M-:' doing some weird thing in the original buffer to trigger
>> this oddity?
>
> Doing M-: in what part of the recipe?

Eval this:

  (progn (switch-to-buffer "test")
         (insert "aline1\nbline2\ncline3\n")
         (put-text-property 7 14 'invisible t) ; region [\nbline2]
         (goto-char (point-min)))

Then

C-e
M-: 4 RET

This will move point to the start of the next line about half the time.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#9034: point adjustment (with invisible text) moves to the next line

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: dmitry.kurochkin@gmail.com,  9034@debbugs.gnu.org
> Date: Thu, 26 Aug 2021 18:33:29 +0200
> 
> Eval this:
> 
>   (progn (switch-to-buffer "test")
>          (insert "aline1\nbline2\ncline3\n")
>          (put-text-property 7 14 'invisible t) ; region [\nbline2]
>          (goto-char (point-min)))
> 
> Then
> 
> C-e
> M-: 4 RET
> 
> This will move point to the start of the next line about half the time.

I cannot reproduce this: no matter how many times I do the "C-e M-:"
dance, point stays put.

bug#9034: point adjustment (with invisible text) moves to the next line

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> I cannot reproduce this: no matter how many times I do the "C-e M-:"
> dance, point stays put.

Does it help if you only do the `C-e' once?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#9034: point adjustment (with invisible text) moves to the next line

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: dmitry.kurochkin@gmail.com,  9034@debbugs.gnu.org
> Date: Thu, 26 Aug 2021 19:05:15 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I cannot reproduce this: no matter how many times I do the "C-e M-:"
> > dance, point stays put.
> 
> Does it help if you only do the `C-e' once?

I tried that as well, to no avail.

I think a more efficient way of understanding what's going on with M-:
is to step through the code in keyboard.c with a debugger.