bug#45319: 28.0.50; Redisplay: `forward-char' moves backwards into invisible text

bug#45319: 28.0.50; Redisplay: `forward-char' moves backwards into invisible text

[Michael Heerdegen]

[-- Attachment #1: Type: text/plain, Size: 282 bytes --]


Hello,

I have found a strange problem I encountered in the Magit Log buffer,
but it is reproducible in any buffer in emacs -Q.  CC'ing Jonas
Bernoulli, maintainer of Magit, for information.

I insert something like this into e.g. *scratch* (looks a bit like a
Magit log buffer):


[-- Attachment #2: test.txt --]
[-- Type: text/plain, Size: 72 bytes --]

11111
  a
  b
  c
22222
  x
  y
  z
33333
  f
  g
  h
44444
  i
  j
  k

[-- Attachment #3: Type: text/plain, Size: 1562 bytes --]


Lets do this:

  (add-hook 'post-command-hook (lambda () (message "%s" (point))))

to see the value of point when we move around in the buffer.

I want to hide lines that start with whitespace, so I do this:

  (defvar my-magit-log-show-only-hash-lines-mode-keywords
    `((,(rx (and bol space (* any) eol))
       (0 (let ((beg (match-beginning 0))
                (end (1+ (match-end 0))))
            (add-text-properties beg end
                                 (list 'invisible 'my-magit-log-show-only-hash-lines))
            nil)))))

  (define-minor-mode my-magit-log-show-only-hash-lines-mode
    "..." nil nil nil
    (if (not my-magit-log-show-only-hash-lines-mode)
        (remove-from-invisibility-spec 'my-magit-log-show-only-hash-lines)
      (font-lock-add-keywords nil my-magit-log-show-only-hash-lines-mode-keywords)
      (add-to-invisibility-spec 'my-magit-log-show-only-hash-lines)))

Now enable that mode in the buffer and M-x font-lock-fontify-buffer.

When I now move around in the buffer with the arrow keys, it often
happens that when point is at the beginning of a visible line, and I hit
<right>, the echo area shows that point actually has moved backwards into
the invisible area somewhere before that line, but the cursor is placed
at the expected position.  Further movement behaves as expected from the
drawn cursor position, just the value of `point' sometimes is
inconsistent, and that can cause trouble (like incorrect section
highlighting in the Magit Log buffer).


TIA,

Michael.

bug#45319: 28.0.50; Redisplay: `forward-char' moves backwards into invisible text

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Date: Sat, 19 Dec 2020 07:10:53 +0100
> Cc: Jonas Bernoulli <jonas@bernoul.li>
> 
>   (defvar my-magit-log-show-only-hash-lines-mode-keywords
>     `((,(rx (and bol space (* any) eol))
>        (0 (let ((beg (match-beginning 0))
>                 (end (1+ (match-end 0))))
>             (add-text-properties beg end
>                                  (list 'invisible 'my-magit-log-show-only-hash-lines))
>             nil)))))
> 
>   (define-minor-mode my-magit-log-show-only-hash-lines-mode
>     "..." nil nil nil
>     (if (not my-magit-log-show-only-hash-lines-mode)
>         (remove-from-invisibility-spec 'my-magit-log-show-only-hash-lines)
>       (font-lock-add-keywords nil my-magit-log-show-only-hash-lines-mode-keywords)
>       (add-to-invisibility-spec 'my-magit-log-show-only-hash-lines)))
> 
> Now enable that mode in the buffer and M-x font-lock-fontify-buffer.
> 
> When I now move around in the buffer with the arrow keys, it often
> happens that when point is at the beginning of a visible line, and I hit
> <right>, the echo area shows that point actually has moved backwards into
> the invisible area somewhere before that line, but the cursor is placed
> at the expected position.  Further movement behaves as expected from the
> drawn cursor position, just the value of `point' sometimes is
> inconsistent, and that can cause trouble (like incorrect section
> highlighting in the Magit Log buffer).

Does setting global-disable-point-adjustment non-nil and repeating
your experiment explain what happens?

It isn't a redisplay problem.

bug#45319: 28.0.50; Redisplay: `forward-char' moves backwards into invisible text

[Michael Heerdegen]

Eli Zaretskii <eliz@gnu.org> writes:

> Does setting global-disable-point-adjustment non-nil and repeating
> your experiment explain what happens?

I currently can't try because I'm eating ATM (yeah, shouldn't read mails
while eating...) but in the Magit buffer that was the first thing I had
tried and it didn't have an effect.

> It isn't a redisplay problem.

Point is moved from a visible into an invisible area - that should not
happen, no?  Even with point-adjustment enabled.

Regards,

Michael.

bug#45319: 28.0.50; Redisplay: `forward-char' moves backwards into invisible text

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Cc: 45319@debbugs.gnu.org,  jonas@bernoul.li
> Date: Sat, 19 Dec 2020 12:21:27 +0100
> 
> > Does setting global-disable-point-adjustment non-nil and repeating
> > your experiment explain what happens?
> 
> I currently can't try because I'm eating ATM (yeah, shouldn't read mails
> while eating...) but in the Magit buffer that was the first thing I had
> tried and it didn't have an effect.

then maybe your recipe is not a complete demonstration what happens in
Magit buffers.

> > It isn't a redisplay problem.
> 
> Point is moved from a visible into an invisible area - that should not
> happen, no?  Even with point-adjustment enabled.

Why not?  "Invisible" means it is not displayed, but it doesn't mean
point cannot be there.  We have point-adjustment to avoid confusing
users because otherwise when point enters invisible text, cursor
movement commands would seem to have no effect, but that doesn't mean
point cannot enter invisible text.

bug#45319: 28.0.50; Redisplay: `forward-char' moves backwards into invisible text

[Michael Heerdegen]

Eli Zaretskii <eliz@gnu.org> writes:

> Why not?  "Invisible" means it is not displayed, but it doesn't mean
> point cannot be there.  We have point-adjustment to avoid confusing
> users because otherwise when point enters invisible text, cursor
> movement commands would seem to have no effect, but that doesn't mean
> point cannot enter invisible text.

But in the situation I described, I have been starting at a visible
position, `forward-char' should move point to the next char which is
also a visible, instead Emacs moves point some characters backwards into
an unrelated invisible position, that makes no sense to me.

Michael.

bug#45319: 28.0.50; Redisplay: `forward-char' moves backwards into invisible text

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Cc: 45319@debbugs.gnu.org,  jonas@bernoul.li
> Date: Sat, 19 Dec 2020 13:09:49 +0100
> 
> But in the situation I described, I have been starting at a visible
> position, `forward-char' should move point to the next char which is
> also a visible, instead Emacs moves point some characters backwards into
> an unrelated invisible position, that makes no sense to me.

You judge what Emacs does with point by a report from a
post-command-hook.  But that's not necessarily what Emacs does with
point: the latter is what is shown by "C-x =".  Does that ever show
moving back?

bug#45319: 28.0.50; Redisplay: `forward-char' moves backwards into invisible text

[Michael Heerdegen]

Eli Zaretskii <eliz@gnu.org> writes:

> You judge what Emacs does with point by a report from a
> post-command-hook.  But that's not necessarily what Emacs does with
> point: the latter is what is shown by "C-x =".  Does that ever show
> moving back?

Magit's section highlighting is performed in post-command-hook, so it
highlights the wrong section in this case based on that value.  Do you
have a solution for this kind of problem?  hl-line-mode has the same
problem btw.

Ok, have tried now with (setq global-disable-point-adjustment t) and it
avoids the problem in my test case, yes.  However it's not what I want
to do in this buffer (or generally in this kind of situation, most of
the time).

Thanks,

Michael.

bug#45319: 28.0.50; Redisplay: `forward-char' moves backwards into invisible text

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Cc: 45319@debbugs.gnu.org,  jonas@bernoul.li
> Date: Sat, 19 Dec 2020 13:44:05 +0100
> 
> Magit's section highlighting is performed in post-command-hook, so it
> highlights the wrong section in this case based on that value.  Do you
> have a solution for this kind of problem?

I don't think I understand the problem well enough yet to answer that
question, especially since you say that your simplified recipe behaves
differently.

What exactly is the problem in the magit case? does it use the value
of point in some way, and if so, how?  IOW, what is "section
highlighting" in magit?

> Ok, have tried now with (setq global-disable-point-adjustment t) and it
> avoids the problem in my test case, yes.  However it's not what I want
> to do in this buffer (or generally in this kind of situation, most of
> the time).

A stab in the dark: there's disable-point-adjustment, which could be
set temporarily, perhaps that could help?

bug#45319: 28.0.50; Redisplay: `forward-char' moves backwards into invisible text

[Michael Heerdegen]

Eli Zaretskii <eliz@gnu.org> writes:

> I don't think I understand the problem well enough yet to answer that
> question, especially since you say that your simplified recipe behaves
> differently.
>
> What exactly is the problem in the magit case? does it use the value
> of point in some way, and if so, how?  IOW, what is "section
> highlighting" in magit?

Let me ask a question first, for my better understanding, I'm curious:
what's your opinion about the (in my eyes) wrong value of point present
when `post-command-hook' is run?  Is it (1) expected and totally makes
sense, should be like that, or more like (2) not relevant, should not be
relied on, or (3) makes not much sense and is not so good but hard to
get to grips?

> A stab in the dark: there's disable-point-adjustment, which could be
> set temporarily, perhaps that could help?

Would that help to fix the behavior of `hl-line-mode' marking the wrong
line?

The Magit case is not much different.  Here a "section" is simply not a
line but a set of consecutive lines denoting a commit (headline + diff,
for example).  As with hl-line-mode, the wrong section is highlighted
when moving.  But since some commands depend on the currently
highlighted section, the consequence is that these commands then operate
on the wrong section.


Regards,

Michael.

bug#45319: 28.0.50; Redisplay: `forward-char' moves backwards into invisible text

[Michael Heerdegen]

Eli Zaretskii <eliz@gnu.org> writes:

> A stab in the dark: there's disable-point-adjustment, which could be
> set temporarily, perhaps that could help?

I see now that I once already had filed a bug report concerning a
similar thing, and one thing Stefan mentioned there is relevant in this
case: with point "at" the beginning of a visible line, after invisible
lines,

  (get-pos-property (point) 'invisible) ==> t

so I'm trying with

  (setf (alist-get 'invisible text-property-default-nonsticky) t)

now.  Seems it helps (for my minor mode at least).


Thanks,

Michael.

bug#45319: 28.0.50; Redisplay: `forward-char' moves backwards into invisible text

[Michael Heerdegen]

Eli Zaretskii <eliz@gnu.org> writes:

> What exactly is the problem in the magit case?

I see now that that problem has been discussed in Bug#23079.  If you
think that everything is alright with my test case you might close this
bug and probably merge with that one.

Thanks,

Michael.

bug#45319: 28.0.50; Redisplay: `forward-char' moves backwards into invisible text

[Eli Zaretskii]

> From: Michael Heerdegen <michael_heerdegen@web.de>
> Cc: 45319@debbugs.gnu.org,  jonas@bernoul.li
> Date: Sun, 20 Dec 2020 09:01:52 +0100
> 
> I see now that that problem has been discussed in Bug#23079.  If you
> think that everything is alright with my test case you might close this
> bug and probably merge with that one.

I closed this one.  Merging them is too much of a hassle with
debbugs's unhelpful defaults for archived bugs.

The only other thing I could say, which may or may not be helpful for
your use case, is that Lisp programs shouldn't assume they will always
find point outside of invisible portion of the text, and instead
actively look for the first visible position after point.