`vertical-motion', `goto-line' set point to invisible text

`vertical-motion', `goto-line' set point to invisible text

[Dmitry Kurochkin]

Hi all.

While working on notmuch [1] emacs client, I stumbled upon an unexpected
behavior: `beginning-of-visual-line' places point to invisible text in
the beginning to the line.  I.e. you have:

  line1
  line2    <--- this line is not visible
  line3    <--- point is on this line

in this case, `beginning-of-visual-line' will set the position to start
of line2 (which is invisible), not line3.  It differs from what
`move-beginning-of-line' and even does not match
`line-beginning-position'.  `beginning-of-visual-line' uses
`vertical-motion' to do the job.  `goto-line' has a similar behavior.  I
believe there are more functions like this.

This does not look right to me.  I expect these functions never set
point inside invisible text and there should be some general way to
protect from this.  But perhaps I am wrong and we have to manually skip
all invisible text forward after any point move?

Here is a small test to demonstrate the issue:

(progn (message "test begin")
       (switch-to-buffer "test")
       (insert "line1\nline2\nline3\n")
       (goto-line 2)
       (put-text-property (line-beginning-position)
                          (line-beginning-position 2)
                          'invisible 'invis1)
       (goto-line 3)
       (message "point #1: %s" (point))
       (add-to-invisibility-spec 'invis1)
       (move-beginning-of-line nil)
       (message "point #2: %s" (point))
       (vertical-motion 0)
       (message "point #3: %s, invisible-p: %s" (point) (invisible-p (point)))
       (message "test end"))

Regards,
  Dmitry

[1] http://notmuchmail.org/

Re: `vertical-motion', `goto-line' set point to invisible text

[Eli Zaretskii]

> From: Dmitry Kurochkin <dmitry.kurochkin@gmail.com>
> Date: Sun, 03 Jul 2011 06:59:49 +0400
> 
> While working on notmuch [1] emacs client, I stumbled upon an unexpected
> behavior: `beginning-of-visual-line' places point to invisible text in
> the beginning to the line.  I.e. you have:
> 
>   line1
>   line2    <--- this line is not visible
>   line3    <--- point is on this line
> 
> in this case, `beginning-of-visual-line' will set the position to start
> of line2 (which is invisible), not line3.  It differs from what
> `move-beginning-of-line' and even does not match
> `line-beginning-position'.  `beginning-of-visual-line' uses
> `vertical-motion' to do the job.  `goto-line' has a similar behavior.  I
> believe there are more functions like this.

See the node "Invisible Text" in the ELisp manual.  Some functions are
explicitly programmed to special behavior in the vicinity of invisible
text, others aren't.

> This does not look right to me.  I expect these functions never set
> point inside invisible text and there should be some general way to
> protect from this.

Are you sure you don't mix invisible with intangible?  Invisible means
just that: not shown on the screen.  It doesn't mean point cannot
enter the invisible portion.  Emacs does try to move point out of
invisible range of text, but it does so in its command loop, _after_
the cursor motion functions and the display engine did their job.  So
your test program just shows that cursor motion has no problem getting
into the invisible region, which I think is not a bug.

> But perhaps I am wrong and we have to manually skip
> all invisible text forward after any point move?

Yes.

There is something strange in how we behave in this example, though.
To see it, modify your test program to make each line's 1st character
different from other lines.  Then, after the program ends, C-a on the
3rd line and type "C-x =": Emacs says the character at point is the
first character of the invisible line 2, which seems wrong, as the
display shows line 3.  In fact, you cannot place point on the 1st
character of line 3: C-b from the 2nd character of line 3 gets you to
the 1st character of line 2!  I think this happens because the range
of invisible characters includes the newline of line 2.

Re: `vertical-motion', `goto-line' set point to invisible text

[Dmitry Kurochkin]

Hi Eli.

On Sun, 03 Jul 2011 02:55:31 -0400, Eli Zaretskii <eliz@gnu.org> wrote:
> > From: Dmitry Kurochkin <dmitry.kurochkin@gmail.com>
> > Date: Sun, 03 Jul 2011 06:59:49 +0400
> > 
> > While working on notmuch [1] emacs client, I stumbled upon an unexpected
> > behavior: `beginning-of-visual-line' places point to invisible text in
> > the beginning to the line.  I.e. you have:
> > 
> >   line1
> >   line2    <--- this line is not visible
> >   line3    <--- point is on this line
> > 
> > in this case, `beginning-of-visual-line' will set the position to start
> > of line2 (which is invisible), not line3.  It differs from what
> > `move-beginning-of-line' and even does not match
> > `line-beginning-position'.  `beginning-of-visual-line' uses
> > `vertical-motion' to do the job.  `goto-line' has a similar behavior.  I
> > believe there are more functions like this.
> 
> See the node "Invisible Text" in the ELisp manual.  Some functions are
> explicitly programmed to special behavior in the vicinity of invisible
> text, others aren't.
> 

Thanks for the hint.

> > This does not look right to me.  I expect these functions never set
> > point inside invisible text and there should be some general way to
> > protect from this.
> 
> Are you sure you don't mix invisible with intangible?

Yes, I know invisible is not the same as intangible.

>  Invisible means
> just that: not shown on the screen.  It doesn't mean point cannot
> enter the invisible portion.  Emacs does try to move point out of
> invisible range of text, but it does so in its command loop, _after_
> the cursor motion functions and the display engine did their job.

I guess that explains why in notmuch (which has multiple lines hidden,
i.e. email body) I hit C-a, then M-: "(point)" I get X, then if I do M-:
"(point)" again, I get Y.  Is there a way to ask Emacs to sync point to
visible position, so that I do not have to do it by hand?

>  So
> your test program just shows that cursor motion has no problem getting
> into the invisible region, which I think is not a bug.
> 
> > But perhaps I am wrong and we have to manually skip
> > all invisible text forward after any point move?
> 
> Yes.
> 

Thanks for your answers.

> There is something strange in how we behave in this example, though.
> To see it, modify your test program to make each line's 1st character
> different from other lines.  Then, after the program ends, C-a on the
> 3rd line and type "C-x =": Emacs says the character at point is the
> first character of the invisible line 2, which seems wrong, as the
> display shows line 3.  In fact, you cannot place point on the 1st
> character of line 3: C-b from the 2nd character of line 3 gets you to
> the 1st character of line 2!  I think this happens because the range
> of invisible characters includes the newline of line 2.

Indeed, I did not notice that when testing in notmuch (I guess because
there are many lines hidden).

`backward-char' steps 1 position as expected.  I.e. M-: (progn
(backward-char) (point)) works fine.  But after that M-: (point) returns
another position which corresponds to the beginning of the hidden line.
I guess this has to do command loop I know nothing about.  IMO Emacs
should not touch the point in this case, because it is already visible.
Sounds like a bug to me.

Regards,
  Dmitry

Re: `vertical-motion', `goto-line' set point to invisible text

[Eli Zaretskii]

> From: Dmitry Kurochkin <dmitry.kurochkin@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Sun, 03 Jul 2011 11:31:23 +0400
> 
> Is there a way to ask Emacs to sync point to visible position, so
> that I do not have to do it by hand?

Not sure what that means.  Can you explain more about what you are
looking for?

> I.e. M-: (progn (backward-char) (point)) works fine.  But after that
> M-: (point) returns another position which corresponds to the
> beginning of the hidden line.  I guess this has to do command loop I
> know nothing about.

When a command finishes and Emacs is idle (i.e. has nothing else to
do), it examines the position of point.  If it finds that point is on
some character where it shouldn't be, it moves point to the next
"suitable" character in the direction of last move.  Text where point
shouldn't be includes: invisible text, non-base character that belongs
to a composed character, and text covered by a "replacing" `display'
property.

> IMO Emacs should not touch the point in this case, because it is
> already visible.  Sounds like a bug to me.

If no one comes up with a plausible explanation until tomorrow, feel
free to submit a bug report.

Btw, I just tried past releases, and Emacs 21.4 didn't have this
problem, but Emacs 22.3 did.  I guess that's a side effect of point
adjustment described above, which Emacs 21.4 didn't have: in Emacs
21.4, the cursor stays put for several C-f or C-b commands, as long as
point is on invisible text.

Re: `vertical-motion', `goto-line' set point to invisible text

[Stefan Monnier]

> While working on notmuch [1] emacs client, I stumbled upon an unexpected
> behavior: `beginning-of-visual-line' places point to invisible text in
> the beginning to the line.  I.e. you have:

>   line1
>   line2    <--- this line is not visible
>   line3    <--- point is on this line

> in this case, `beginning-of-visual-line' will set the position to start
> of line2 (which is invisible), not line3.  It differs from what
> `move-beginning-of-line' and even does not match
> `line-beginning-position'.  `beginning-of-visual-line' uses
> `vertical-motion' to do the job.  `goto-line' has a similar behavior.  I
> believe there are more functions like this.

> This does not look right to me.  I expect these functions never set
> point inside invisible text and there should be some general way to
> protect from this.

AFAIK none of those positions are *inside* invisible text (remember
that point is always between two chars).


        Stefan

Re: `vertical-motion', `goto-line' set point to invisible text

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Sun, 03 Jul 2011 10:50:21 -0400
> Cc: emacs-devel@gnu.org
> 
> AFAIK none of those positions are *inside* invisible text

Try "M-x describe-text-properties RET", and you will see this isn't
true.

Re: `vertical-motion', `goto-line' set point to invisible text

[Dmitry Kurochkin]

On Sun, 03 Jul 2011 16:33:54 +0300, Eli Zaretskii <eliz@gnu.org> wrote:
> > From: Dmitry Kurochkin <dmitry.kurochkin@gmail.com>
> > Cc: emacs-devel@gnu.org
> > Date: Sun, 03 Jul 2011 11:31:23 +0400
> > 
> > Is there a way to ask Emacs to sync point to visible position, so
> > that I do not have to do it by hand?
> 
> Not sure what that means.  Can you explain more about what you are
> looking for?
> 

I am looking for a function that does something like what you described
below.  I.e. moves point to the next suitable character in the given
direction.  At the moment I do it with smth like:

  ;; if point is invisible, skip forward to visible text
  (while (invisible-p p)
    (setq p (next-single-char-property-change p 'invisible)))

> > I.e. M-: (progn (backward-char) (point)) works fine.  But after that
> > M-: (point) returns another position which corresponds to the
> > beginning of the hidden line.  I guess this has to do command loop I
> > know nothing about.
> 
> When a command finishes and Emacs is idle (i.e. has nothing else to
> do), it examines the position of point.  If it finds that point is on
> some character where it shouldn't be, it moves point to the next
> "suitable" character in the direction of last move.  Text where point
> shouldn't be includes: invisible text, non-base character that belongs
> to a composed character, and text covered by a "replacing" `display'
> property.
> 

Thanks for the detailed explanation.

> > IMO Emacs should not touch the point in this case, because it is
> > already visible.  Sounds like a bug to me.
> 
> If no one comes up with a plausible explanation until tomorrow, feel
> free to submit a bug report.
> 

ok

> Btw, I just tried past releases, and Emacs 21.4 didn't have this
> problem, but Emacs 22.3 did.  I guess that's a side effect of point
> adjustment described above, which Emacs 21.4 didn't have: in Emacs
> 21.4, the cursor stays put for several C-f or C-b commands, as long as
> point is on invisible text.

I will put a reference in the bug report.

Regards,
  Dmitry

Re: `vertical-motion', `goto-line' set point to invisible text

[Stefan Monnier]

>> AFAIK none of those positions are *inside* invisible text
> Try "M-x describe-text-properties RET", and you will see this isn't
> true.

AFAIK, describe-text-properties describes the properties of the char
*after* point, so if you're at the beginning of invisible text (yet not
inside it), describe-text-properties will show the `invisible' property.


        Stefan

Re: `vertical-motion', `goto-line' set point to invisible text

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: dmitry.kurochkin@gmail.com,  emacs-devel@gnu.org
> Date: Mon, 04 Jul 2011 10:01:10 -0400
> 
> >> AFAIK none of those positions are *inside* invisible text
> > Try "M-x describe-text-properties RET", and you will see this isn't
> > true.
> 
> AFAIK, describe-text-properties describes the properties of the char
> *after* point, so if you're at the beginning of invisible text (yet not
> inside it), describe-text-properties will show the `invisible' property.

So how many character positions are invisible after this:

       (insert "line1\nline2\nline3\n")
       (goto-line 2)
       (put-text-property (line-beginning-position)
                          (line-beginning-position 2)
                          'invisible 'invis1)

?

Also, since we show cursor on the character after point, which is
invisible, what exactly do we mean to achieve in this case by
adjust_point_for_property?

Finally, what do you think of this:

  (progn (switch-to-buffer "test")
	 (insert "aline1\nbline2\ncline3\n")
	 (goto-line 2)
	 (put-text-property (line-beginning-position)
			    (line-beginning-position 2)
			    'invisible 'invis1)
	 (add-to-invisibility-spec 'invis1)
	 (goto-char (point-max)))

Eval this in *scratch*, then type "C-p C-x =".  You will see that
Emacs reports that point is position 8 and the character at point is
`b', whereas what is shown (correctly) under the cursor is `c' whose
buffer position is 15.  Do you think this is correct behavior?

Re: `vertical-motion', `goto-line' set point to invisible text

[Stefan Monnier]

>> >> AFAIK none of those positions are *inside* invisible text
>> > Try "M-x describe-text-properties RET", and you will see this isn't
>> > true.
>> AFAIK, describe-text-properties describes the properties of the char
>> *after* point, so if you're at the beginning of invisible text (yet not
>> inside it), describe-text-properties will show the `invisible' property.

> So how many character positions are invisible after this:
>        (insert "line1\nline2\nline3\n")
>        (goto-line 2)
>        (put-text-property (line-beginning-position)
>                           (line-beginning-position 2)
>                           'invisible 'invis1)
> ?

Without running the code, I'd say 6 ("line2\n").  Why?

> Also, since we show cursor on the character after point,

By default, yes.

> which is invisible, what exactly do we mean to achieve in this case by
> adjust_point_for_property?

The intention is just to avoid having point in the middle of invisible
text and have things like C-f and C-b appear to do nothing (because
they only move within some invisible text chunk).
Also it's so that inserting text will usually result in this text being
visible (and hopefully inserted right where the cursor was displayed).

None of those properties are guaranteed by adjust_point_for_property,
sadly, but it's the motivation behind it.

> Finally, what do you think of this:

>   (progn (switch-to-buffer "test")
> 	 (insert "aline1\nbline2\ncline3\n")
> 	 (goto-line 2)
> 	 (put-text-property (line-beginning-position)
> 			    (line-beginning-position 2)
> 			    'invisible 'invis1)
> 	 (add-to-invisibility-spec 'invis1)
> 	 (goto-char (point-max)))

> Eval this in *scratch*, then type "C-p C-x =".  You will see that
> Emacs reports that point is position 8 and the character at point is
> `b', whereas what is shown (correctly) under the cursor is `c' whose
> buffer position is 15.  Do you think this is correct behavior?

In this area, I don't think there's a clear cut definition of what is
correct and what is not.  Depending on what is being done some behavior
is preferable, and in other cases another behavior is preferable.
If point is anywhere between "bline2\n" the display will be identical,
so the fact that the cursor is drawn over the "c" that follows it does
not necessarily imply that point should be "at the end of "bline2\n",
since the cursor is also drawn right after the "aline1\n" and by that
logic point should be at the beginning of "bline2\n".

What tilts the decision one way rather than the other here is the
stickiness: for all positions other than right before "bline2\n",
self-insert-command at point will result in an invisible char being
inserted, which is very rarely the intention of the user.  Of course, in
a read-only buffer this consideration may not matter, but note that this
stickiness issue also happens to be a way by which Elisp packages can
control in which part of the invisible text point will end up.


        Stefan

Re: `vertical-motion', `goto-line' set point to invisible text

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: dmitry.kurochkin@gmail.com,  emacs-devel@gnu.org
> Date: Mon, 04 Jul 2011 14:10:11 -0400
> 
> >> >> AFAIK none of those positions are *inside* invisible text
> >> > Try "M-x describe-text-properties RET", and you will see this isn't
> >> > true.
> >> AFAIK, describe-text-properties describes the properties of the char
> >> *after* point, so if you're at the beginning of invisible text (yet not
> >> inside it), describe-text-properties will show the `invisible' property.
> 
> > So how many character positions are invisible after this:
> >        (insert "line1\nline2\nline3\n")
> >        (goto-line 2)
> >        (put-text-property (line-beginning-position)
> >                           (line-beginning-position 2)
> >                           'invisible 'invis1)
> > ?
> 
> Without running the code, I'd say 6 ("line2\n").  Why?

Because with your reasoning it's only 5.

> >   (progn (switch-to-buffer "test")
> > 	 (insert "aline1\nbline2\ncline3\n")
> > 	 (goto-line 2)
> > 	 (put-text-property (line-beginning-position)
> > 			    (line-beginning-position 2)
> > 			    'invisible 'invis1)
> > 	 (add-to-invisibility-spec 'invis1)
> > 	 (goto-char (point-max)))
> 
> > Eval this in *scratch*, then type "C-p C-x =".  You will see that
> > Emacs reports that point is position 8 and the character at point is
> > `b', whereas what is shown (correctly) under the cursor is `c' whose
> > buffer position is 15.  Do you think this is correct behavior?
> 
> In this area, I don't think there's a clear cut definition of what is
> correct and what is not.

??? How could having a cursor on `c' and reporting it on `b' be
"correct" by any measure?

> If point is anywhere between "bline2\n" the display will be identical,
> so the fact that the cursor is drawn over the "c" that follows it does
> not necessarily imply that point should be "at the end of "bline2\n",
> since the cursor is also drawn right after the "aline1\n" and by that
> logic point should be at the beginning of "bline2\n".

But what about the fact that you cannot have point at position 15,
which is outside the invisible portion?  How can that be anything but
a bug?

Re: `vertical-motion', `goto-line' set point to invisible text

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: dmitry.kurochkin@gmail.com,  emacs-devel@gnu.org
> Date: Mon, 04 Jul 2011 14:10:11 -0400
> 
> >> >> AFAIK none of those positions are *inside* invisible text
> >> > Try "M-x describe-text-properties RET", and you will see this isn't
> >> > true.
> >> AFAIK, describe-text-properties describes the properties of the char
> >> *after* point, so if you're at the beginning of invisible text (yet not
> >> inside it), describe-text-properties will show the `invisible' property.
> 
> > So how many character positions are invisible after this:
> >        (insert "line1\nline2\nline3\n")
> >        (goto-line 2)
> >        (put-text-property (line-beginning-position)
> >                           (line-beginning-position 2)
> >                           'invisible 'invis1)
> > ?
> 
> Without running the code, I'd say 6 ("line2\n").  Why?

Because with your reasoning it's only 5.

> >   (progn (switch-to-buffer "test")
> > 	 (insert "aline1\nbline2\ncline3\n")
> > 	 (goto-line 2)
> > 	 (put-text-property (line-beginning-position)
> > 			    (line-beginning-position 2)
> > 			    'invisible 'invis1)
> > 	 (add-to-invisibility-spec 'invis1)
> > 	 (goto-char (point-max)))
> 
> > Eval this in *scratch*, then type "C-p C-x =".  You will see that
> > Emacs reports that point is position 8 and the character at point is
> > `b', whereas what is shown (correctly) under the cursor is `c' whose
> > buffer position is 15.  Do you think this is correct behavior?
> 
> In this area, I don't think there's a clear cut definition of what is
> correct and what is not.

??? How could having a cursor on `c' and reporting it on `b' be
"correct" by any measure?

> If point is anywhere between "bline2\n" the display will be identical,
> so the fact that the cursor is drawn over the "c" that follows it does
> not necessarily imply that point should be "at the end of "bline2\n",
> since the cursor is also drawn right after the "aline1\n" and by that
> logic point should be at the beginning of "bline2\n".

But what about the fact that you cannot have point at position 15,
which is outside the invisible portion?  How can that be anything but
a bug?

Re: `vertical-motion', `goto-line' set point to invisible text

[Dmitry Kurochkin]

On Mon, 04 Jul 2011 14:10:11 -0400, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> >> >> AFAIK none of those positions are *inside* invisible text
> >> > Try "M-x describe-text-properties RET", and you will see this isn't
> >> > true.
> >> AFAIK, describe-text-properties describes the properties of the char
> >> *after* point, so if you're at the beginning of invisible text (yet not
> >> inside it), describe-text-properties will show the `invisible' property.
> 
> > So how many character positions are invisible after this:
> >        (insert "line1\nline2\nline3\n")
> >        (goto-line 2)
> >        (put-text-property (line-beginning-position)
> >                           (line-beginning-position 2)
> >                           'invisible 'invis1)
> > ?
> 
> Without running the code, I'd say 6 ("line2\n").  Why?
> 
> > Also, since we show cursor on the character after point,
> 
> By default, yes.
> 
> > which is invisible, what exactly do we mean to achieve in this case by
> > adjust_point_for_property?
> 
> The intention is just to avoid having point in the middle of invisible
> text and have things like C-f and C-b appear to do nothing (because
> they only move within some invisible text chunk).
> Also it's so that inserting text will usually result in this text being
> visible (and hopefully inserted right where the cursor was displayed).
> 
> None of those properties are guaranteed by adjust_point_for_property,
> sadly, but it's the motivation behind it.
> 
> > Finally, what do you think of this:
> 
> >   (progn (switch-to-buffer "test")
> > 	 (insert "aline1\nbline2\ncline3\n")
> > 	 (goto-line 2)
> > 	 (put-text-property (line-beginning-position)
> > 			    (line-beginning-position 2)
> > 			    'invisible 'invis1)
> > 	 (add-to-invisibility-spec 'invis1)
> > 	 (goto-char (point-max)))
> 
> > Eval this in *scratch*, then type "C-p C-x =".  You will see that
> > Emacs reports that point is position 8 and the character at point is
> > `b', whereas what is shown (correctly) under the cursor is `c' whose
> > buffer position is 15.  Do you think this is correct behavior?
> 
> In this area, I don't think there's a clear cut definition of what is
> correct and what is not.  Depending on what is being done some behavior
> is preferable, and in other cases another behavior is preferable.
> If point is anywhere between "bline2\n" the display will be identical,
> so the fact that the cursor is drawn over the "c" that follows it does
> not necessarily imply that point should be "at the end of "bline2\n",
> since the cursor is also drawn right after the "aline1\n" and by that
> logic point should be at the beginning of "bline2\n".
> 
> What tilts the decision one way rather than the other here is the
> stickiness: for all positions other than right before "bline2\n",
> self-insert-command at point will result in an invisible char being
> inserted, which is very rarely the intention of the user.  Of course, in
> a read-only buffer this consideration may not matter, but note that this
> stickiness issue also happens to be a way by which Elisp packages can
> control in which part of the invisible text point will end up.
> 

That makes sense.  Thank you.

Perhaps in case of notmuch the best way is to hide the preceding newline
character instead of the trailing one.  This way there will always be a
visible newline between different messages and cursor-moving functions
(e.g. `vertical-motion') should not move beyond it.

Still looking for an answer to the following question earlier in this
thread:

    I am looking for a function that does something like what you described
    below.  I.e. moves point to the next suitable character in the given
    direction.  At the moment I do it with smth like:

      ;; if point is invisible, skip forward to visible text
      (while (invisible-p p)
        (setq p (next-single-char-property-change p 'invisible)))

Regards,
  Dmitry

> 
>         Stefan

Re: `vertical-motion', `goto-line' set point to invisible text

[Eli Zaretskii]

> From: Dmitry Kurochkin <dmitry.kurochkin@gmail.com>
> Date: Tue, 05 Jul 2011 06:22:56 +0400
> Cc: emacs-devel@gnu.org
> 
> Still looking for an answer to the following question earlier in this
> thread:
> 
>     I am looking for a function that does something like what you described
>     below.  I.e. moves point to the next suitable character in the given
>     direction.  At the moment I do it with smth like:
> 
>       ;; if point is invisible, skip forward to visible text
>       (while (invisible-p p)
>         (setq p (next-single-char-property-change p 'invisible)))

I think your code is the right way.  I don't think anything
ready-to-use exists.

Re: `vertical-motion', `goto-line' set point to invisible text

[Dmitry Kurochkin]

On Tue, 05 Jul 2011 06:22:56 +0400, Dmitry Kurochkin <dmitry.kurochkin@gmail.com> wrote:
> On Mon, 04 Jul 2011 14:10:11 -0400, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> > >> >> AFAIK none of those positions are *inside* invisible text
> > >> > Try "M-x describe-text-properties RET", and you will see this isn't
> > >> > true.
> > >> AFAIK, describe-text-properties describes the properties of the char
> > >> *after* point, so if you're at the beginning of invisible text (yet not
> > >> inside it), describe-text-properties will show the `invisible' property.
> > 
> > > So how many character positions are invisible after this:
> > >        (insert "line1\nline2\nline3\n")
> > >        (goto-line 2)
> > >        (put-text-property (line-beginning-position)
> > >                           (line-beginning-position 2)
> > >                           'invisible 'invis1)
> > > ?
> > 
> > Without running the code, I'd say 6 ("line2\n").  Why?
> > 
> > > Also, since we show cursor on the character after point,
> > 
> > By default, yes.
> > 
> > > which is invisible, what exactly do we mean to achieve in this case by
> > > adjust_point_for_property?
> > 
> > The intention is just to avoid having point in the middle of invisible
> > text and have things like C-f and C-b appear to do nothing (because
> > they only move within some invisible text chunk).
> > Also it's so that inserting text will usually result in this text being
> > visible (and hopefully inserted right where the cursor was displayed).
> > 
> > None of those properties are guaranteed by adjust_point_for_property,
> > sadly, but it's the motivation behind it.
> > 
> > > Finally, what do you think of this:
> > 
> > >   (progn (switch-to-buffer "test")
> > > 	 (insert "aline1\nbline2\ncline3\n")
> > > 	 (goto-line 2)
> > > 	 (put-text-property (line-beginning-position)
> > > 			    (line-beginning-position 2)
> > > 			    'invisible 'invis1)
> > > 	 (add-to-invisibility-spec 'invis1)
> > > 	 (goto-char (point-max)))
> > 
> > > Eval this in *scratch*, then type "C-p C-x =".  You will see that
> > > Emacs reports that point is position 8 and the character at point is
> > > `b', whereas what is shown (correctly) under the cursor is `c' whose
> > > buffer position is 15.  Do you think this is correct behavior?
> > 
> > In this area, I don't think there's a clear cut definition of what is
> > correct and what is not.  Depending on what is being done some behavior
> > is preferable, and in other cases another behavior is preferable.
> > If point is anywhere between "bline2\n" the display will be identical,
> > so the fact that the cursor is drawn over the "c" that follows it does
> > not necessarily imply that point should be "at the end of "bline2\n",
> > since the cursor is also drawn right after the "aline1\n" and by that
> > logic point should be at the beginning of "bline2\n".
> > 
> > What tilts the decision one way rather than the other here is the
> > stickiness: for all positions other than right before "bline2\n",
> > self-insert-command at point will result in an invisible char being
> > inserted, which is very rarely the intention of the user.  Of course, in
> > a read-only buffer this consideration may not matter, but note that this
> > stickiness issue also happens to be a way by which Elisp packages can
> > control in which part of the invisible text point will end up.
> > 
> 
> That makes sense.  Thank you.
> 
> Perhaps in case of notmuch the best way is to hide the preceding newline
> character instead of the trailing one.  This way there will always be a
> visible newline between different messages and cursor-moving functions
> (e.g. `vertical-motion') should not move beyond it.
> 

While trying this, I found another issue with how emacs adjusts the
point when it is idle.  We have 3 lines and the second one is hidden.
But now we hide the newline on line1 instead of line2:

  (progn (switch-to-buffer "test")
         (insert "aline1\nbline2\ncline3\n")
         (put-text-property 7 14 'invisible t)
         (goto-char (point-min)))

Now if you run M-: (progn (end-of-visual-line) (point)), it moves to
point 14, which is end of line 2 as expected.  But then emacs adjusts
the point and moves it to beginning of line3, C-x = says point is 15.

Also, global-hl-line-mode does not work correctly for line1.  Just go to
the start of buffer and move forward to see some weird behavior.

Regards,
  Dmitry

> Still looking for an answer to the following question earlier in this
> thread:
> 
>     I am looking for a function that does something like what you described
>     below.  I.e. moves point to the next suitable character in the given
>     direction.  At the moment I do it with smth like:
> 
>       ;; if point is invisible, skip forward to visible text
>       (while (invisible-p p)
>         (setq p (next-single-char-property-change p 'invisible)))
> 
> Regards,
>   Dmitry
> 
> > 
> >         Stefan

Re: `vertical-motion', `goto-line' set point to invisible text

[Eli Zaretskii]

> From: Dmitry Kurochkin <dmitry.kurochkin@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Tue, 05 Jul 2011 07:52:09 +0400
> 
>   (progn (switch-to-buffer "test")
>          (insert "aline1\nbline2\ncline3\n")
>          (put-text-property 7 14 'invisible t)
>          (goto-char (point-min)))
> 
> Now if you run M-: (progn (end-of-visual-line) (point)), it moves to
> point 14, which is end of line 2 as expected.  But then emacs adjusts
> the point and moves it to beginning of line3, C-x = says point is 15.

Interestingly, the first time you do this, there's no adjustment from
14 to 15.  Sounds like some snafu with the last point position?

I can confirm that this happens because point adjustment: setting
global-disable-point-adjustment to non-nil eliminates the problem:
point stays at position 14.

Also, C-e behaves correctly no matter what.  So it's something
specific to end-of-visual-line or vertical-motion.

Perhaps stepping through adjust_point_for_property with a debugger
will show what's wrong.

Re: `vertical-motion', `goto-line' set point to invisible text

[Dmitry Kurochkin]

On Tue, 05 Jul 2011 03:56:55 -0400, Eli Zaretskii <eliz@gnu.org> wrote:
> > From: Dmitry Kurochkin <dmitry.kurochkin@gmail.com>
> > Cc: emacs-devel@gnu.org
> > Date: Tue, 05 Jul 2011 07:52:09 +0400
> > 
> >   (progn (switch-to-buffer "test")
> >          (insert "aline1\nbline2\ncline3\n")
> >          (put-text-property 7 14 'invisible t)
> >          (goto-char (point-min)))
> > 
> > Now if you run M-: (progn (end-of-visual-line) (point)), it moves to
> > point 14, which is end of line 2 as expected.  But then emacs adjusts
> > the point and moves it to beginning of line3, C-x = says point is 15.
> 
> Interestingly, the first time you do this, there's no adjustment from
> 14 to 15.  Sounds like some snafu with the last point position?
> 
> I can confirm that this happens because point adjustment: setting
> global-disable-point-adjustment to non-nil eliminates the problem:
> point stays at position 14.
> 
> Also, C-e behaves correctly no matter what.  So it's something
> specific to end-of-visual-line or vertical-motion.
> 
> Perhaps stepping through adjust_point_for_property with a debugger
> will show what's wrong.

I have opened a bug report #9034 [1] for this issue.  Hopefully it would
get some attention.  Seems it should not be too hard to investigate
investigate the problem for somebody how understands the code well
enough.

Please provide additional details for the bug report if I missed
anything.

Regards,
  Dmitry

[1] http://debbugs.gnu.org/cgi/bugreport.cgi?bug=9034