bug#19188: point adjustemnt moves *into* invisible text

bug#19188: point adjustemnt moves *into* invisible text

[Jonas Bernoulli]

When moving backward "point adjustment" which is supposed to move
point out of an invisible region may end up doing the opposite.

This problem exists at least in 24.3 and 24.4.

1. Yank this in an empty buffer in fundamental-mode and evaluate it.

(progn
  (goto-char (point-min))
  (insert "1\n"
	  (propertize "3\n" 'invisible t)
	  "5\n"
	  "7\n")
	  (backward-char 2))

2. The cursor is now on the "7", which also is the 7th character.

   The buffer looks like
   .-----
   |1
   |5
   |7
   |(progn
   |...
   `-----

   The cursor sits on the "7" and

     M-: (point) => 7
   
3. Move to "5" using e.g. C-p or C-b C-b.

   The cursor is now on "5", which also is the 5th character.

   However point is not were the cursor is

     M-: (point) => 3

The problem is in the code that is supposed to move point *out* of an
invisible region, does the opposite when moving backward places point
on the first character after an invisible region.  It moves to the
beginning of the preceding invisible region.

When point adjustment is disabled (non-nil disable-point-adjustment or
global-disable-point-adjustment) then this does not happen.

It also does not happen when moving forward, e.g. starting at "1"
C-p C-f places the cursor on "5" *and* point is also 5.

bug#19188: point adjustemnt moves *into* invisible text

[Stefan Monnier]

tags 19188 notabug
thanks

>    However point is not were the cursor is
>      M-: (point) => 3
> The problem is in the code that is supposed to move point *out* of an
> invisible region, does the opposite when moving backward places point
> on the first character after an invisible region.  It moves to the
> beginning of the preceding invisible region.

That is a common misunderstanding.  The fact that point is equal to
3 means that point is *between* character 2 and character 3.  So it's not
*inside* an invisible text, but is right at the boundary.

The position 5 (i.e. between character 4 and character 5) is at the
other end of the boundary.

The reason why Emacs decided to put point at position 3 rather than
leave it at position 5 is because the boundary at position 3 is "less
invisible" than the boundary at position 5.
You can check it with

  M-: (list (get-pos-property 3 'invisible) (get-pos-property 5 'invisible)) RET

This is because text-properties by default are front-non-sticky and
rear-sticky, so if point is at position 5 and you type a character, that
character will inherit the invisible property, whereas if you're at
position 3 and you type a character this character will not inherit the
invisible property.

If you want point to be at position 5 rather than position 3, then you
need to change the front/rear-stickiness of this invisible
property accordingly.

> When point adjustment is disabled (non-nil disable-point-adjustment or
> global-disable-point-adjustment) then this does not happen.

I assume you know why ;-)

> It also does not happen when moving forward, e.g. starting at "1"
> C-p C-f places the cursor on "5" *and* point is also 5.

C-p C-f doesn't do it for me (it doesn't even reach the invisible part of the
text), and if I change the recipe to C-f C-f it doesn't work either
(point stays at position 3).

But indeed C-n gets me to position 5, which is wrong (and doing M-: (point)
returns 5 but moves me to position 3, so doing it again returns 3 :-( ).
So we do have a bug here.


        Stefan

bug#19188: point adjustemnt moves *into* invisible text

[Jonas Bernoulli]

Sorry for the delay.

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

> The fact that point is equal to 3 means that point is *between*
> character 2 and character 3.  So it's not *inside* an invisible text,
> but is right at the boundary.

I understand that and should have chosen my words more carefully.

I noticed that the behavior was not consistent and jumped to the
conclusion that the invisibility property did receive some special
handling but that there was a bug in how that was done.  Turns out it is
supposed to behave according to its stickyness, like other properties.

It is however possible to get the behavior that I want.  This requires
that `invisible's stickyness is set accordingly.

  (push (cons 'invisible t) text-property-default-nonsticky)

For modes with collapsible section (magit, org-mode ...) this behavior
makes a lot of sense.  Basically this teaches Emacs to define
`invisible' as "cannot be seen by the eye", which happens to be a
reasonable definition I would think :-)

Also required is this hack which relies on `get-text-property' and
`get-pos-property' disagreeing about the value of `invisible'.

  (add-hook 'post-command-hook #'magit-post-command-adjust-point t t)

  (defun magit-post-command-adjust-point ()
    (when (and (get-text-property (point) 'invisible)
               (not (if (fboundp 'get-pos-property) ; since 24.4, see #1671
                        (get-pos-property (point) 'invisible)
                      (get-text-property (1+ (point)) 'invisible))))
      (goto-char (next-single-char-property-change (point) 'invisible))))

Is there a better way?

I am not (no longer) convinced that Emacs should necessarily behave the
way I thought it was supposed to; not by default at least.  But I do
think that in some cases it feels strange to adjust point, just because
point is "property-invisible" and even though the following character
were the cursor would be displayed is "visible-to-the-eye".  It is
especially confusing when only point is adjusted and the cursor stays
were it would have been without the adjustment of point.

So maybe, just maybe, it might make sense if "property-visible-point"
and "eye-visible-cursor" would agree by default.




>
> The position 5 (i.e. between character 4 and character 5) is at the
> other end of the boundary.
>
> The reason why Emacs decided to put point at position 3 rather than
> leave it at position 5 is because the boundary at position 3 is "less
> invisible" than the boundary at position 5.
> You can check it with
>
>   M-: (list (get-pos-property 3 'invisible) (get-pos-property 5 'invisible)) RET
>
> This is because text-properties by default are front-non-sticky and
> rear-sticky, so if point is at position 5 and you type a character, that
> character will inherit the invisible property, whereas if you're at
> position 3 and you type a character this character will not inherit the
> invisible property.
>
> If you want point to be at position 5 rather than position 3, then you
> need to change the front/rear-stickiness of this invisible
> property accordingly.
>
>> When point adjustment is disabled (non-nil disable-point-adjustment or
>> global-disable-point-adjustment) then this does not happen.
>
> I assume you know why ;-)
>
>> It also does not happen when moving forward, e.g. starting at "1"
>> C-p C-f places the cursor on "5" *and* point is also 5.
>
> C-p C-f doesn't do it for me (it doesn't even reach the invisible part of the
> text), and if I change the recipe to C-f C-f it doesn't work either
> (point stays at position 3).
>
> But indeed C-n gets me to position 5, which is wrong (and doing M-: (point)
> returns 5 but moves me to position 3, so doing it again returns 3 :-( ).
> So we do have a bug here.
>
>
>         Stefan

bug#19188: point adjustemnt moves *into* invisible text

[Stefan Monnier]

> Also required is this hack which relies on `get-text-property' and
> `get-pos-property' disagreeing about the value of `invisible'.

>   (add-hook 'post-command-hook #'magit-post-command-adjust-point t t)

>   (defun magit-post-command-adjust-point ()
>     (when (and (get-text-property (point) 'invisible)
>                (not (if (fboundp 'get-pos-property) ; since 24.4, see #1671
>                         (get-pos-property (point) 'invisible)
>                       (get-text-property (1+ (point)) 'invisible))))
>       (goto-char (next-single-char-property-change (point) 'invisible))))

> Is there a better way?

I don't know, because I don't know what it's trying to do.

> It is especially confusing when only point is adjusted and the cursor
> stays were it would have been without the adjustment of point.

IIUC you're describing a situation where the display is incorrect, so
I think this would be a bug.

>> But indeed C-n gets me to position 5, which is wrong (and doing M-: (point)
>> returns 5 but moves me to position 3, so doing it again returns 3 :-( ).
>> So we do have a bug here.

I filed a separate bug report about this problem.


        Stefan