How to solve overlay conflict between invisible and display?

[Tassilo Horn <tsdh@gnu.org>]

Hi all,

I have a problem where two modes use overlays in a way where the result
is not desirable.  One mode hides complete sections of text by adding
overlays with `invisible` property, the other mode alters the displayed
text using overlays with `display` property.

In the concrete scenario [1], the invisible-overlays always surround the
overlays with display property, e.g., the latter are nested in the
former.

  [invisible t [display foo] [display bar]...]

So effectively the nested overlays have both a non-nil `invisible`
property and additionally a `display` property.  In that case, it seems
the `display` property wins.  That can be validated using this simple
demo command in a buffer with text where the word foo will get an
overlay with both `invisible` and `display` property:

--8<---------------cut here---------------start------------->8---
(defun th/overlay-invisible-and-display-test ()
  (interactive)
  (save-excursion
    (goto-char 1)
    (while (re-search-forward "foo" nil t)
      (let ((o (make-overlay (match-beginning 0) (match-end 0))))
        (overlay-put o 'invisible t)
        (overlay-put o 'display "§")))))
--8<---------------cut here---------------end--------------->8---

In my scenario, I would like that the outer invisible overlay also hides
the nested display overlays.

In there anything either the mode producing the outer `invisible`
overlay or the mode producing the inner `display` overlay could do so
that what should be invisible actually becomes invisible?

I've already tried setting `display` to nil in the outer overlays and
raising the priority in the howe that `display nil` then overrides
`display "some string"` from the inner overlays but that doesn't seem to
make any difference.

Well, it sounds like common sense that invisible should win over
display, or to be more clear, display should only be effective on
visible overlays.  So maybe that's actually a bug in emacs (current
master)?

Thanks,
  Tassilo

[1] https://github.com/magit/magit/issues/5176