bug#21710: 25.0.50; self-insert-command before a folded outline inserts after the outline

bug#21710: 25.0.50; self-insert-command before a folded outline inserts after the outline

[Oleh Krehel]

To reproduce with "emacs -Q", create a file ~/outline-test with the
following contents:

* foo
some text
* bar
some text

Open the file, move to start, "M-x" `outline-minor-mode',
"M-x" `outline-hide-sublevels'.

Move to the end of bar and insert "tt" - it works correctly.

But move to the end of foo and insert "tt": here's what it looks like
("|" is the point):

* foo...t|
* bartt...

Here's the full text after "M-x" `outline-show-all':

* foot
some textt
* bartt
some text

As you see, the second "t" was inserted after the outline.
Since Org mode uses outlines, this bug also appears there.

bug#21710: 25.0.50; self-insert-command before a folded outline inserts after the outline

[Eli Zaretskii]

> From: Oleh Krehel <ohwoeowho@gmail.com>
> Date: Mon, 19 Oct 2015 17:16:11 +0200
> 
> To reproduce with "emacs -Q", create a file ~/outline-test with the
> following contents:
> 
> * foo
> some text
> * bar
> some text
> 
> Open the file, move to start, "M-x" `outline-minor-mode',
> "M-x" `outline-hide-sublevels'.
> 
> Move to the end of bar and insert "tt" - it works correctly.
> 
> But move to the end of foo and insert "tt": here's what it looks like
> ("|" is the point):
> 
> * foo...t|
> * bartt...
> 
> Here's the full text after "M-x" `outline-show-all':
> 
> * foot
> some textt
> * bartt
> some text
> 
> As you see, the second "t" was inserted after the outline.

The reason for this is that on all header lines but the last inserting
the first 't' makes it invisible.  On the last line, 't' stays
visible.  Then point adjustment kicks in, and moves point out of the
invisible region, so the second 't' is inserted at the end of the
sub-level text.

This hints at stickiness, but I cannot find anything in the
documentation about stickiness of overlays.

bug#21710: 25.0.50; self-insert-command before a folded outline inserts after the outline

[Oleh Krehel]

Eli Zaretskii <eliz@gnu.org> writes:

> The reason for this is that on all header lines but the last inserting
> the first 't' makes it invisible.

This isn't a good behavior. `self-insert-command' should not put text
into an invisible region. Instead it should put it ahead of it.

"C-e" should complement this behavior, starting from:

* |foo...
* bar...

pressing "C-e" should result in:

* foo|...
* bar...

And from the state above, pressing "C-d" should result in:
(user-error "Overlay read-only"), instead of deleting hidden characters.

Actually, the points above are often a source of confusion and
frustration. I wouldn't be surprised that this bug was reported before.
Currently, I work around this problem either by unfolding all outlines
while I'm editing, or transforming "foo|" into "foot|" with "C-b ot C-d"
instead of a simple "t".

> Then point adjustment kicks in, and moves point out of the
> invisible region, so the second 't' is inserted at the end of the
> sub-level text.

Can you point me to a place where I can read up on this point
adjustment? I'd like to help in fixing this bug, if it's not too
complicated. So far, I'm looking at this code:

(defun outline-flag-region (from to flag)
  "Hide or show lines from FROM to TO, according to FLAG.
If FLAG is nil then text is shown, while if FLAG is t the text is hidden."
  (remove-overlays from to 'invisible 'outline)
  (when flag
    ;; We use `front-advance' here because the invisible text begins at the
    ;; very end of the heading, before the newline, so text inserted at FROM
    ;; belongs to the heading rather than to the entry.
    (let ((o (make-overlay from to nil 'front-advance)))
      (overlay-put o 'evaporate t)
      (overlay-put o 'invisible 'outline)
      (overlay-put o 'isearch-open-invisible
		   (or outline-isearch-open-invisible-function
		       'outline-isearch-open-invisible))))
  ;; Seems only used by lazy-lock.  I.e. obsolete.
  (run-hooks 'outline-view-change-hook))

The comment about `front-advance' implies that what I proposed above was
the intended behavior, which is broken currently.

bug#21710: 25.0.50; self-insert-command before a folded outline inserts after the outline

[Eli Zaretskii]

> From: Oleh Krehel <ohwoeowho@gmail.com>
> Cc: 21710@debbugs.gnu.org
> Date: Wed, 21 Oct 2015 15:05:16 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > The reason for this is that on all header lines but the last inserting
> > the first 't' makes it invisible.
> 
> This isn't a good behavior. `self-insert-command' should not put text
> into an invisible region.

It doesn't.  It inserts the character at point.  The problem is that
the inserted character becomes invisible because the overlay becomes
applicable to it.

> "C-e" should complement this behavior, starting from:
> 
> * |foo...
> * bar...
> 
> pressing "C-e" should result in:
> 
> * foo|...
> * bar...

If that's what you want, you need to program that into the mode.  The
default effect of C-e is to put point one character before the newline
on the same visible line, which is _after_ the invisible text.  IOW,
this is not a bug, but perhaps a missing feature.

> And from the state above, pressing "C-d" should result in:
> (user-error "Overlay read-only"), instead of deleting hidden characters.

Once again, you need to program that.  By default, C-d just deletes
the character at point.

> > Then point adjustment kicks in, and moves point out of the
> > invisible region, so the second 't' is inserted at the end of the
> > sub-level text.
> 
> Can you point me to a place where I can read up on this point
> adjustment?

Some minimal Lisp-level documentation is in "(elisp) Adjusting Point.
For the gory details, see the function adjust_point_for_property in
keyboard.c.

> I'd like to help in fixing this bug, if it's not too complicated.

It's not a bug in point adjustment, strictly speaking: that part does
its job as intended.  The problem is with the fact that the inserted
character becomes invisible.

> So far, I'm looking at this code:
> 
> (defun outline-flag-region (from to flag)
>   "Hide or show lines from FROM to TO, according to FLAG.
> If FLAG is nil then text is shown, while if FLAG is t the text is hidden."
>   (remove-overlays from to 'invisible 'outline)
>   (when flag
>     ;; We use `front-advance' here because the invisible text begins at the
>     ;; very end of the heading, before the newline, so text inserted at FROM
>     ;; belongs to the heading rather than to the entry.
>     (let ((o (make-overlay from to nil 'front-advance)))
>       (overlay-put o 'evaporate t)
>       (overlay-put o 'invisible 'outline)
>       (overlay-put o 'isearch-open-invisible
> 		   (or outline-isearch-open-invisible-function
> 		       'outline-isearch-open-invisible))))
>   ;; Seems only used by lazy-lock.  I.e. obsolete.
>   (run-hooks 'outline-view-change-hook))
> 
> The comment about `front-advance' implies that what I proposed above was
> the intended behavior, which is broken currently.

Then why does it work correctly on the last heading?

bug#21710: 25.0.50; self-insert-command before a folded outline inserts after the outline

[Lars Ingebrigtsen]

Oleh Krehel <ohwoeowho@gmail.com> writes:

> To reproduce with "emacs -Q", create a file ~/outline-test with the
> following contents:
>
> * foo
> some text
> * bar
> some text
>
> Open the file, move to start, "M-x" `outline-minor-mode',
> "M-x" `outline-hide-sublevels'.
>
> Move to the end of bar and insert "tt" - it works correctly.
>
> But move to the end of foo and insert "tt": here's what it looks like
> ("|" is the point):
>
> * foo...t|
> * bartt...
>
> Here's the full text after "M-x" `outline-show-all':
>
> * foot
> some textt
> * bartt
> some text

(I'm going through old bug reports that unfortunately weren't
resolved at the time.)

I'm not seeing quite what you're seeing here in Emacs 26.1 or 28.  I'm getting:

* foo...ttttt|
* bar...

And then after showing, I get:

* foo
some texttttttt
* bar
some text

So this works better than the described problem (no "bartt"), but it's
still somewhat confusing.

On the other hand, I'm not sure quite what it's supposed to do if you
type after the dots.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#21710: 25.0.50; self-insert-command before a folded outline inserts after the outline

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Sun, 15 Aug 2021 19:01:42 +0200
> Cc: 21710@debbugs.gnu.org
> 
> Oleh Krehel <ohwoeowho@gmail.com> writes:
> 
> > To reproduce with "emacs -Q", create a file ~/outline-test with the
> > following contents:
> >
> > * foo
> > some text
> > * bar
> > some text
> >
> > Open the file, move to start, "M-x" `outline-minor-mode',
> > "M-x" `outline-hide-sublevels'.
> >
> > Move to the end of bar and insert "tt" - it works correctly.
> >
> > But move to the end of foo and insert "tt": here's what it looks like
> > ("|" is the point):
> >
> > * foo...t|
> > * bartt...
> >
> > Here's the full text after "M-x" `outline-show-all':
> >
> > * foot
> > some textt
> > * bartt
> > some text
> 
> (I'm going through old bug reports that unfortunately weren't
> resolved at the time.)
> 
> I'm not seeing quite what you're seeing here in Emacs 26.1 or 28.  I'm getting:
> 
> * foo...ttttt|
> * bar...

I get in Emacs 28 exactly what the OP describes.

But isn't that expected, given the way outline-minor-mode uses the
invisible property?

> On the other hand, I'm not sure quite what it's supposed to do if you
> type after the dots.

He was typing _before_ the dots.  He said "move to the end of foo".

bug#21710: 25.0.50; self-insert-command before a folded outline inserts after the outline

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> I get in Emacs 28 exactly what the OP describes.

[...]

>> On the other hand, I'm not sure quite what it's supposed to do if you
>> type after the dots.
>
> He was typing _before_ the dots.  He said "move to the end of foo".

Oh, before the dots...

OK, hide the sublevels and then hit "t" twice.  I get:

* foo...t|
* bar...

Unhiding:

* foot
some textt
* bar
some text

So I'm not getting the "bartt"?

> But isn't that expected, given the way outline-minor-mode uses the
> invisible property?

Probably, but it's not ... good?  I have a vague feeling the correct
pre-advance/post-advance (or whatever they're called) properties on the
"..." could fix some of this.  But it's been a long time since I last
played with invisible text.

Perhaps it would be worthwhile to experiment with putting the `display'
property over the hidden text (with "...") instead and see whether that
has fewer quirks.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#21710: 25.0.50; self-insert-command before a folded outline inserts after the outline

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: ohwoeowho@gmail.com,  21710@debbugs.gnu.org
> Date: Sun, 15 Aug 2021 20:12:10 +0200
> 
> OK, hide the sublevels and then hit "t" twice.  I get:
> 
> * foo...t|
> * bar...
> 
> Unhiding:
> 
> * foot
> some textt
> * bar
> some text
> 
> So I'm not getting the "bartt"?

In my testing, all the headings but the last exhibit the "foo"
behavior.  So it's the last heading that is the odd one out.