How to solve overlay conflict between invisible and display?

How to solve overlay conflict between invisible and display?

[Tassilo Horn]

Hi all,

I have a problem where two modes use overlays in a way where the result
is not desirable.  One mode hides complete sections of text by adding
overlays with `invisible` property, the other mode alters the displayed
text using overlays with `display` property.

In the concrete scenario [1], the invisible-overlays always surround the
overlays with display property, e.g., the latter are nested in the
former.

  [invisible t [display foo] [display bar]...]

So effectively the nested overlays have both a non-nil `invisible`
property and additionally a `display` property.  In that case, it seems
the `display` property wins.  That can be validated using this simple
demo command in a buffer with text where the word foo will get an
overlay with both `invisible` and `display` property:

--8<---------------cut here---------------start------------->8---
(defun th/overlay-invisible-and-display-test ()
  (interactive)
  (save-excursion
    (goto-char 1)
    (while (re-search-forward "foo" nil t)
      (let ((o (make-overlay (match-beginning 0) (match-end 0))))
        (overlay-put o 'invisible t)
        (overlay-put o 'display "§")))))
--8<---------------cut here---------------end--------------->8---

In my scenario, I would like that the outer invisible overlay also hides
the nested display overlays.

In there anything either the mode producing the outer `invisible`
overlay or the mode producing the inner `display` overlay could do so
that what should be invisible actually becomes invisible?

I've already tried setting `display` to nil in the outer overlays and
raising the priority in the howe that `display nil` then overrides
`display "some string"` from the inner overlays but that doesn't seem to
make any difference.

Well, it sounds like common sense that invisible should win over
display, or to be more clear, display should only be effective on
visible overlays.  So maybe that's actually a bug in emacs (current
master)?

Thanks,
  Tassilo

[1] https://github.com/magit/magit/issues/5176

Re: How to solve overlay conflict between invisible and display?

[Eli Zaretskii]

> From: Tassilo Horn <tsdh@gnu.org>
> Date: Wed, 17 Jul 2024 10:16:01 +0200
> 
> I have a problem where two modes use overlays in a way where the result
> is not desirable.  One mode hides complete sections of text by adding
> overlays with `invisible` property, the other mode alters the displayed
> text using overlays with `display` property.

That's not a good combination.

> In the concrete scenario [1], the invisible-overlays always surround the
> overlays with display property, e.g., the latter are nested in the
> former.
> 
>   [invisible t [display foo] [display bar]...]
> 
> So effectively the nested overlays have both a non-nil `invisible`
> property and additionally a `display` property.  In that case, it seems
> the `display` property wins.

What do you mean by "nested"?  If the invisible property starts
_before_ the display property, the invisible property "wins".  This
variant of your text program:

(defun th/overlay-invisible-and-display-test ()
  (interactive)
  (save-excursion
    (goto-char 1)
    (while (re-search-forward "foo" nil t)
      (let ((o1 (make-overlay (match-beginning 0) (match-end 0)))
	    (o2 (make-overlay (1+ (match-beginning 0)) (1- (match-end 0)))))
        (overlay-put o1 'invisible t)
        (overlay-put o2 'display "§")))))

when run from *scratch*, completely removes "foo" from display,
showing only the quotes, which means the invisible property "won".

> That can be validated using this simple
> demo command in a buffer with text where the word foo will get an
> overlay with both `invisible` and `display` property:
> 
> --8<---------------cut here---------------start------------->8---
> (defun th/overlay-invisible-and-display-test ()
>   (interactive)
>   (save-excursion
>     (goto-char 1)
>     (while (re-search-forward "foo" nil t)
>       (let ((o (make-overlay (match-beginning 0) (match-end 0))))
>         (overlay-put o 'invisible t)
>         (overlay-put o 'display "§")))))
> --8<---------------cut here---------------end--------------->8---

Here, both invisible and display properties start at the same buffer
position, so the display property "wins".  Why? simply because the
display code examines the properties sequentially, and it examines the
display property _before_ the invisible property.  IOW, this is the
side effect of how properties are handled by the display engine.

> In my scenario, I would like that the outer invisible overlay also hides
> the nested display overlays.

Then make the invisible property start before display.

> In there anything either the mode producing the outer `invisible`
> overlay or the mode producing the inner `display` overlay could do so
> that what should be invisible actually becomes invisible?

Short of deleting the overlay with the display property, I don't think
you can do anything other than changing where the invisible property
begins.

> I've already tried setting `display` to nil in the outer overlays and
> raising the priority in the howe that `display nil` then overrides
> `display "some string"` from the inner overlays but that doesn't seem to
> make any difference.

That's because display property with value nil is the same as no
display property.

> Well, it sounds like common sense that invisible should win over
> display, or to be more clear, display should only be effective on
> visible overlays.  So maybe that's actually a bug in emacs (current
> master)?

The display property _replaces_ the text it covers, so it also kind-of
"hides".  Which one "hides" first is what determines the results.  And
no, it is not a bug: Emacs always worked like that.  To be sure, I've
ran your recipe in Emacs 21.4, and got the same result.

Re: How to solve overlay conflict between invisible and display?

[Tassilo Horn]

Eli Zaretskii <eliz@gnu.org> writes:

> What do you mean by "nested"?  If the invisible property starts
> _before_ the display property, the invisible property "wins".  This
> variant of your text program:
>
> (defun th/overlay-invisible-and-display-test ()
>   (interactive)
>   (save-excursion
>     (goto-char 1)
>     (while (re-search-forward "foo" nil t)
>       (let ((o1 (make-overlay (match-beginning 0) (match-end 0)))
> 	    (o2 (make-overlay (1+ (match-beginning 0)) (1- (match-end 0)))))
>         (overlay-put o1 'invisible t)
>         (overlay-put o2 'display "§")))))
>
> when run from *scratch*, completely removes "foo" from display,
> showing only the quotes, which means the invisible property "won".

Ok, I see.  I assumed that when both overlays start xor end at the same
position, that would still count as "one is nested in the other".

>> In my scenario, I would like that the outer invisible overlay also
>> hides the nested display overlays.
>
> Then make the invisible property start before display.

I guess the invisible property overlay could start one char earlier
(covering the newline before) and additionally add a before-string
property with that newline.  I'll try that.

Thanks a lot,
  Tassilo