bug#31067: 27.0.50; After-string hidden by subsequent invisible text

bug#31067: 27.0.50; After-string hidden by subsequent invisible text

[Stefan Monnier]

Package: Emacs
Version: 27.0.50


In a fresh Emacs session define:

    (defun foo ()
      (interactive)
      (with-current-buffer "*scratch*"
        (add-to-invisibility-spec '(foo . t))
        (let ((beg (point)))
          (insert "text")
          (let ((ol1 (make-overlay beg (point))))
            (overlay-put ol1 'after-string "!after!")
            (overlay-put ol1 'evaporate t)))
        (let ((beg (point)))
          (insert "\nhidden")
          (let ((ol1 (make-overlay beg (point))))
            (overlay-put ol1 'invisible 'foo)
            (overlay-put ol1 'evaporate t)))))

and then try it with `M-x foo`.
The result for me is to display "text...", whereas I would have expected
"text!after!...".

I.e. the after-string which I placed over "text" gets hidden by
`invisible` property of the overlay placed on the immediately following
"\nhidden".

I know these kinds of interactions are pretty delicate and often
somewhat arbitrary (several behaviors are valid and we just have to pick
one), but I think in this case it is clearly an error.


        Stefan

bug#31067: 27.0.50; After-string hidden by subsequent invisible text

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Wed, 04 Apr 2018 22:28:59 -0400
> 
>     (defun foo ()
>       (interactive)
>       (with-current-buffer "*scratch*"
>         (add-to-invisibility-spec '(foo . t))
>         (let ((beg (point)))
>           (insert "text")
>           (let ((ol1 (make-overlay beg (point))))
>             (overlay-put ol1 'after-string "!after!")
>             (overlay-put ol1 'evaporate t)))
>         (let ((beg (point)))
>           (insert "\nhidden")
>           (let ((ol1 (make-overlay beg (point))))
>             (overlay-put ol1 'invisible 'foo)
>             (overlay-put ol1 'evaporate t)))))
> 
> and then try it with `M-x foo`.
> The result for me is to display "text...", whereas I would have expected
> "text!after!...".
> 
> I.e. the after-string which I placed over "text" gets hidden by
> `invisible` property of the overlay placed on the immediately following
> "\nhidden".
> 
> I know these kinds of interactions are pretty delicate and often
> somewhat arbitrary (several behaviors are valid and we just have to pick
> one), but I think in this case it is clearly an error.

Would you expect the after-string to be shown in the variant below?

(defun foo ()
  (interactive)
  (with-current-buffer "*scratch*"
    (add-to-invisibility-spec '(foo . t))
    (let ((beg (point))
	  end)
      (insert "hidden")
      (setq end (point))
      (insert "text")
      (let ((ol1 (make-overlay beg end)))
        (overlay-put ol1 'after-string "!after!")
        (overlay-put ol1 'evaporate t))
      (let ((ol2 (make-overlay (1+ beg) (point))))
        (overlay-put ol2 'invisible 'foo)
        (overlay-put ol2 'evaporate t)))))

IOW, the question is what should happen when the end-point of the
overlay with after-string is in invisible text?

bug#31067: 27.0.50; After-string hidden by subsequent invisible text

[Stefan Monnier]

> Would you expect the after-string to be shown in the variant below?
>
> (defun foo ()
>   (interactive)
>   (with-current-buffer "*scratch*"
>     (add-to-invisibility-spec '(foo . t))
>     (let ((beg (point))
> 	  end)
>       (insert "hidden")
>       (setq end (point))
>       (insert "text")
>       (let ((ol1 (make-overlay beg end)))
>         (overlay-put ol1 'after-string "!after!")
>         (overlay-put ol1 'evaporate t))
>       (let ((ol2 (make-overlay (1+ beg) (point))))
>         (overlay-put ol2 'invisible 'foo)
>         (overlay-put ol2 'evaporate t)))))
>
> IOW, the question is what should happen when the end-point of the
> overlay with after-string is in invisible text?

If some (or all) of the end of the overlay-with-after-string is made
invisible, then the situation is much less clear and I could see
arguments either way, but if I get to choose then I think it makes sense
to consider that the after string is "attached" to the end of the
overlay, i.e. if the end of the overlay is invisible then so is the
after-string.


        Stefan

bug#31067: 27.0.50; After-string hidden by subsequent invisible text

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: 31067@debbugs.gnu.org
> Date: Thu, 05 Apr 2018 08:23:30 -0400
> 
> > IOW, the question is what should happen when the end-point of the
> > overlay with after-string is in invisible text?
> 
> If some (or all) of the end of the overlay-with-after-string is made
> invisible, then the situation is much less clear and I could see
> arguments either way, but if I get to choose then I think it makes sense
> to consider that the after string is "attached" to the end of the
> overlay, i.e. if the end of the overlay is invisible then so is the
> after-string.

But that's exactly what happens in your original example.

Btw, "some or all of the end" is a strange wording, because the end is
a (dimensionless) point.

bug#31067: 27.0.50; After-string hidden by subsequent invisible text

[Stefan Monnier]

>> > IOW, the question is what should happen when the end-point of the
>> > overlay with after-string is in invisible text?
>> If some (or all) of the end of the overlay-with-after-string is made
>> invisible, then the situation is much less clear and I could see
>> arguments either way, but if I get to choose then I think it makes sense
>> to consider that the after string is "attached" to the end of the
>> overlay, i.e. if the end of the overlay is invisible then so is the
>> after-string.
> But that's exactly what happens in your original example.

Hmmm.... not that I can see: the overlay covers "text" and none of it
is hidden.

I guess you could pay attention to the stickiness of the boundaries, but
in my example the end of the overlay-with-after-string is not sticky, so
you could say that it ends right after "t" and not on the immediately
following "(dimensionless) point".  Also I changed my test so that the
beginning of the invisible overlay is not sticky (so that the
"(dimensionless) point" between "t" and "\n" is supposedly not affected
by this overlay):

(defun foo ()
  (interactive)
  (with-current-buffer "*scratch*"
    (add-to-invisibility-spec '(foo . t))
    (let ((beg (point)))
      (insert "text")
      (let ((ol1 (make-overlay beg (point))))
        (overlay-put ol1 'after-string "!after!")
        (overlay-put ol1 'evaporate t)))
    (let ((beg (point)))
      (insert "\nhidden")
      (let ((ol1 (make-overlay beg (point) nil t)))
        (overlay-put ol1 'invisible 'foo)
        (overlay-put ol1 'evaporate t)))))

but the result is still the same.  And think this one is even more
clearly a bug, because according to stickiness the two overlays "don't
touch" (as can be tested by carefully moving point right after the "t" and
inserting "-" which gives you a display of "text!after!-" showing that
the "-" was inserted between the two overlays rather than into the
first or into the second or into both).

> Btw, "some or all of the end" is a strange wording,

Indeed, I added "or all" after the fact and did it poorly.  I meant "if
the last few chars covered by the overlay (or the whole text covered
by the overlay) is made invisible ...".


        Stefan

bug#31067: 27.0.50; After-string hidden by subsequent invisible text

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: 31067@debbugs.gnu.org
> Date: Thu, 05 Apr 2018 11:55:43 -0400
> 
> >> If some (or all) of the end of the overlay-with-after-string is made
> >> invisible, then the situation is much less clear and I could see
> >> arguments either way, but if I get to choose then I think it makes sense
> >> to consider that the after string is "attached" to the end of the
> >> overlay, i.e. if the end of the overlay is invisible then so is the
> >> after-string.
> > But that's exactly what happens in your original example.
> 
> Hmmm.... not that I can see: the overlay covers "text" and none of it
> is hidden.

The overlay's end point is _after_ "text", and that's exactly where
the invisible text starts.  And after-string _follows_ the end of
"text", so it starts where the invisible text starts.

> I guess you could pay attention to the stickiness of the boundaries

I don't think stickiness has anything to do with this.  I could
explain how the particular implementation of these features causes the
after-string to be skipped in this case, but I prefer that we first
establish the principles and the concepts.  From my POV, the
implementation does what it does because it considers after-string,
conceptually, to _follow_ the end-point of the overlay, and in this
case what follows it is invisible.

> And think this one is even more
> clearly a bug, because according to stickiness the two overlays "don't
> touch" (as can be tested by carefully moving point right after the "t" and
> inserting "-" which gives you a display of "text!after!-" showing that
> the "-" was inserted between the two overlays rather than into the
> first or into the second or into both).

They cannot "not touch", because there's nothing between 'after
"text"' and 'before the following point'.  The fact that inserting a
character behaves in some specific way doesn't matter, because the
display engine doesn't (and shouldn't) consider stickiness, it only
considers which display elements follow which.

> I meant "if the last few chars covered by the overlay (or the whole
> text covered by the overlay) is made invisible ...".

And that's what happens: the overlay's end-point is made invisible.

bug#31067: 27.0.50; After-string hidden by subsequent invisible text

[Stefan Monnier]

>> >> If some (or all) of the end of the overlay-with-after-string is made
>> >> invisible, then the situation is much less clear and I could see
>> >> arguments either way, but if I get to choose then I think it makes sense
>> >> to consider that the after string is "attached" to the end of the
>> >> overlay, i.e. if the end of the overlay is invisible then so is the
>> >> after-string.
>> > But that's exactly what happens in your original example.
>> Hmmm.... not that I can see: the overlay covers "text" and none of it
>> is hidden.
> The overlay's end point is _after_ "text", and that's exactly where
> the invisible text starts.  And after-string _follows_ the end of
> "text", so it starts where the invisible text starts.

Yes, but that doesn't mean that the second overlay *covers* the end of
the first.  Whether/when we consider it to cover is something we get
to decide.

Basically, we have:

                      end-of-ol1
                   /              \
    character "t", - after-string - character "\n"
                   \              /
                     start-of-el2

where the middle 3-way thingy has 0 width and we get to treat them as
occurring in any order we like, to some extent (including in several
orders at the same time depending on context).

Currently, it seems that the ordering chosen in this specific example is
something like

   A) character "t", end-of-ol1, start-of-ol2, after-string, character "\n"
or
   B) character "t", start-of-ol2, end-of-ol1, after-string, character "\n"
or
   C) character "t", start-of-ol2, after-string, end-of-ol1, character "\n"

I.e. start-of-ol2 is taken to occur before the after-string, which is
why the after-string is made invisible.

>> I guess you could pay attention to the stickiness of the boundaries
> I don't think stickiness has anything to do with this.

I'm not saying it explains the current behavior, no.  I'm talking about
what kind of model/semantics we could use to justify the current
behavior, and conclude that indeed stickiness can't be used to
justify it.

> I could explain how the particular implementation of these features
> causes the after-string to be skipped in this case, but I prefer that
> we first establish the principles and the concepts.

Agreed.

> The fact that inserting a character behaves in some specific way
> doesn't matter, because the display engine doesn't (and shouldn't)
> consider stickiness, it only considers which display elements
> follow which.

We've used stickiness in various related areas (e.g. in cursor movement)
to try and give some control to the programmer about what should happen
in those borderline cases.

To tell you the truth, I'm not really arguing for the use of
stickiness here.  I'd be perfectly happy with a rule "if the last char
covered by the overlay is visible, then the after-string is also
visible" (which is what I meant by "attached to the end").

>> I meant "if the last few chars covered by the overlay (or the whole
>> text covered by the overlay) is made invisible ...".
> And that's what happens: the overlay's end-point is made invisible.

I said "last few chars", not "end-point".


        Stefan

bug#31067: 27.0.50; After-string hidden by subsequent invisible text

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: 31067@debbugs.gnu.org
> Date: Thu, 05 Apr 2018 15:15:29 -0400
> 
> >> Hmmm.... not that I can see: the overlay covers "text" and none of it
> >> is hidden.
> > The overlay's end point is _after_ "text", and that's exactly where
> > the invisible text starts.  And after-string _follows_ the end of
> > "text", so it starts where the invisible text starts.
> 
> Yes, but that doesn't mean that the second overlay *covers* the end of
> the first.  Whether/when we consider it to cover is something we get
> to decide.

You are talking about "covering" here, and I think this discloses the
mental bias in understanding how before- and after-strings work.
Unlike most other overlay properties, they do not supply any
attributes to the characters "covered" by the overlay.  Instead, they
have effect only at two places: the beginning and the end points of
the overlay.  Consequently, we check for them only when we are about
to display something at these two positions.  And in this case, the
end point is invisible, so we never check any after-strings there.

And yes, this is due to the order of checking for various display
features: invisibility is tested before the overlay strings.  But
there's a good reason for that order, and "fixing" this dubious use
case, should we decide doing that, will probably be messy due to the
need to avoid displaying the same overlay string twice.  So I suggest
that we instead accept this as deliberate and correct behavior.

> Currently, it seems that the ordering chosen in this specific example is
> something like
> 
>    A) character "t", end-of-ol1, start-of-ol2, after-string, character "\n"
> or
>    B) character "t", start-of-ol2, end-of-ol1, after-string, character "\n"
> or
>    C) character "t", start-of-ol2, after-string, end-of-ol1, character "\n"
> 
> I.e. start-of-ol2 is taken to occur before the after-string, which is
> why the after-string is made invisible.

We don't test for start-of-ol2, we test whether text at that position
is invisible, for whatever reason (it could be a text property or an
overlay property).

> I'd be perfectly happy with a rule "if the last char covered by the
> overlay is visible, then the after-string is also visible" (which is
> what I meant by "attached to the end").

I tried to explain above why thinking about "covered by" is wrong when
overlay strings are involved.

bug#31067: 27.0.50; After-string hidden by subsequent invisible text

[Stefan Monnier]

>> >> Hmmm.... not that I can see: the overlay covers "text" and none of it
>> >> is hidden.
>> > The overlay's end point is _after_ "text", and that's exactly where
>> > the invisible text starts.  And after-string _follows_ the end of
>> > "text", so it starts where the invisible text starts.
>> Yes, but that doesn't mean that the second overlay *covers* the end of
>> the first.  Whether/when we consider it to cover is something we get
>> to decide.
> You are talking about "covering" here, and I think this discloses the
> mental bias in understanding how before- and after-strings work.
> Unlike most other overlay properties, they do not supply any
> attributes to the characters "covered" by the overlay.

Note that I was not talking about "after-string covering something" but
about "the invisibility overlay covering the after-string".  For the
`invisible` property, "covering" is very much meaningful, AFAIK.

> And yes, this is due to the order of checking for various display
> features: invisibility is tested before the overlay strings.  But
> there's a good reason for that order, and "fixing" this dubious use
> case, should we decide doing that, will probably be messy due to the
> need to avoid displaying the same overlay string twice.  So I suggest
> that we instead accept this as deliberate and correct behavior.

Here's my use-case:
I have an org-like mode where I add a kind of "summary" of the content
of each section as an after-string on the header (think of it as the
number of sub-sections or the number of words).  This works great as
long as that section is not folded, but as soon as I fold it, the
summary disappears (along with the body, but that part is clearly
intended).  This summary is handy when the section is unfolded but it
would be even more useful when the section is folded.

I wouldn't want to insert those as actual buffer text because they
shouldn't be saved, so it would be a hassle to tweak the save and
auto-save machinery to remove those.

So I'd prefer to keep this as an "unfixed bug".

>> I'd be perfectly happy with a rule "if the last char covered by the
>> overlay is visible, then the after-string is also visible" (which is
>> what I meant by "attached to the end").
> I tried to explain above why thinking about "covered by" is wrong when
> overlay strings are involved.

I'm not talking about the string covering something, I'm talking about
the string's overlay.  IIUC what you're saying is that the connection
between the two is difficult to recover, in the current code.


        Stefan

bug#31067: 27.0.50; After-string hidden by subsequent invisible text

[Eli Zaretskii]

> From: Stefan Monnier <monnier@IRO.UMontreal.CA>
> Cc: 31067@debbugs.gnu.org
> Date: Fri, 06 Apr 2018 08:28:38 -0400
> 
> > You are talking about "covering" here, and I think this discloses the
> > mental bias in understanding how before- and after-strings work.
> > Unlike most other overlay properties, they do not supply any
> > attributes to the characters "covered" by the overlay.
> 
> Note that I was not talking about "after-string covering something" but
> about "the invisibility overlay covering the after-string".  For the
> `invisible` property, "covering" is very much meaningful, AFAIK.

Indeed.  The problem here is that the invisibility overlay covers the
position where displaying the after-string should be considered.

> Here's my use-case:
> I have an org-like mode where I add a kind of "summary" of the content
> of each section as an after-string on the header (think of it as the
> number of sub-sections or the number of words).  This works great as
> long as that section is not folded, but as soon as I fold it, the
> summary disappears (along with the body, but that part is clearly
> intended).  This summary is handy when the section is unfolded but it
> would be even more useful when the section is folded.
> 
> I wouldn't want to insert those as actual buffer text because they
> shouldn't be saved, so it would be a hassle to tweak the save and
> auto-save machinery to remove those.
> 
> So I'd prefer to keep this as an "unfixed bug".

Fine with me (I still hope that leaving such issues will some day
attract volunteers who'd like to do their share of hacking the display
code).

Alternatively, you could, either:

 . separate the invisible text and the after-string with some
   character, or
 . use the 'invisible' text property instead of an overlay

> > I tried to explain above why thinking about "covered by" is wrong when
> > overlay strings are involved.
> 
> I'm not talking about the string covering something, I'm talking about
> the string's overlay.  IIUC what you're saying is that the connection
> between the two is difficult to recover, in the current code.

No, I'm saying that the only connection is the 2 end-points of the
overlay.  The fact that some text is or isn't in-between is
irrelevant.  Talking about "covered text" in this case is detrimental
to understanding how the feature works, because it tends to force us
to think about after-string as being in some sense "related" to the
"covered" text.  It isn't.

bug#31067: 27.0.50; After-string hidden by subsequent invisible text

[Drew Adams]

> You are talking about "covering" here, and I think this discloses the
> mental bias in understanding how before- and after-strings work.
> Unlike most other overlay properties, they do not supply any
> attributes to the characters "covered" by the overlay.  Instead, they
> have effect only at two places: the beginning and the end points of
> the overlay.  Consequently, we check for them only when we are about
> to display something at these two positions.  And in this case, the
> end point is invisible, so we never check any after-strings there.

Sounds like that information is important for a user's mental
model.  Is it presented in the manual?  If not, perhaps it
should be.

bug#31067: 27.0.50; After-string hidden by subsequent invisible text

[Eli Zaretskii]

> Date: Fri, 6 Apr 2018 15:39:45 +0000 (UTC)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: 31067@debbugs.gnu.org
> 
> Sounds like that information is important for a user's mental
> model.  Is it presented in the manual?  If not, perhaps it
> should be.

It will only make sense to document this if there's an agreement that
this is the correct behavior.  It doesn't sound we have such an
agreement at this time.