Re: How to solve overlay conflict between invisible and display?

[Eli Zaretskii <eliz@gnu.org>]

> From: Tassilo Horn <tsdh@gnu.org>
> Date: Wed, 17 Jul 2024 10:16:01 +0200
> 
> I have a problem where two modes use overlays in a way where the result
> is not desirable.  One mode hides complete sections of text by adding
> overlays with `invisible` property, the other mode alters the displayed
> text using overlays with `display` property.

That's not a good combination.

> In the concrete scenario [1], the invisible-overlays always surround the
> overlays with display property, e.g., the latter are nested in the
> former.
> 
>   [invisible t [display foo] [display bar]...]
> 
> So effectively the nested overlays have both a non-nil `invisible`
> property and additionally a `display` property.  In that case, it seems
> the `display` property wins.

What do you mean by "nested"?  If the invisible property starts
_before_ the display property, the invisible property "wins".  This
variant of your text program:

(defun th/overlay-invisible-and-display-test ()
  (interactive)
  (save-excursion
    (goto-char 1)
    (while (re-search-forward "foo" nil t)
      (let ((o1 (make-overlay (match-beginning 0) (match-end 0)))
	    (o2 (make-overlay (1+ (match-beginning 0)) (1- (match-end 0)))))
        (overlay-put o1 'invisible t)
        (overlay-put o2 'display "§")))))

when run from *scratch*, completely removes "foo" from display,
showing only the quotes, which means the invisible property "won".

> That can be validated using this simple
> demo command in a buffer with text where the word foo will get an
> overlay with both `invisible` and `display` property:
> 
> --8<---------------cut here---------------start------------->8---
> (defun th/overlay-invisible-and-display-test ()
>   (interactive)
>   (save-excursion
>     (goto-char 1)
>     (while (re-search-forward "foo" nil t)
>       (let ((o (make-overlay (match-beginning 0) (match-end 0))))
>         (overlay-put o 'invisible t)
>         (overlay-put o 'display "§")))))
> --8<---------------cut here---------------end--------------->8---

Here, both invisible and display properties start at the same buffer
position, so the display property "wins".  Why? simply because the
display code examines the properties sequentially, and it examines the
display property _before_ the invisible property.  IOW, this is the
side effect of how properties are handled by the display engine.

> In my scenario, I would like that the outer invisible overlay also hides
> the nested display overlays.

Then make the invisible property start before display.

> In there anything either the mode producing the outer `invisible`
> overlay or the mode producing the inner `display` overlay could do so
> that what should be invisible actually becomes invisible?

Short of deleting the overlay with the display property, I don't think
you can do anything other than changing where the invisible property
begins.

> I've already tried setting `display` to nil in the outer overlays and
> raising the priority in the howe that `display nil` then overrides
> `display "some string"` from the inner overlays but that doesn't seem to
> make any difference.

That's because display property with value nil is the same as no
display property.

> Well, it sounds like common sense that invisible should win over
> display, or to be more clear, display should only be effective on
> visible overlays.  So maybe that's actually a bug in emacs (current
> master)?

The display property _replaces_ the text it covers, so it also kind-of
"hides".  Which one "hides" first is what determines the results.  And
no, it is not a bug: Emacs always worked like that.  To be sure, I've
ran your recipe in Emacs 21.4, and got the same result.