bug#22320: Overlays with an 'invisible property break stacking of overlay faces

[Eli Zaretskii <eliz@gnu.org>]

> From: Clément Pit--Claudel <clement.pitclaudel@live.com>
> Date: Wed, 6 Jan 2016 13:04:17 -0500
> 
> Adding an 'invisible property to an overlay seems to break stacking of faces.
> To reproduce, run the following snippet. The whole buffer should have a red
> background, but the invisible section does not inherit that background.
> 
> (with-current-buffer (get-buffer-create "bug")
>   (erase-buffer)
>   (fundamental-mode)
>   (add-to-invisibility-spec '(outline . t))
>   (insert "line1 line2 line3")
>   (let ((ov (make-overlay 7 12)))
>     (overlay-put ov 'invisible 'outline))
>   (let ((ov (make-overlay 7 8)))
>     (overlay-put ov 'face '(:underline t)))
>   (let ((ov (make-overlay (point-min) (point-max))))
>     (overlay-put ov 'face '(:background "red")))
>   (pop-to-buffer (current-buffer)))
> 
> I initially thought this was an issue of priorities, but removing the overlay 
> with
> the invisible property makes the issue disappear. Similarly, removing the
> underline overlay makes the issue disappear.

It's a feature.  More accurately, it's the best solution found (more
than 10 years ago!) for a difficult problem.  In a nutshell, in a
general use case the invisible text can (and many times does) have
non-trivial face attributes, and using those faces for displaying the
ellipsis would result in each instance of the ellipsis to have some
random face.  This "feature" makes that face predictable: when the
invisible text has a face that is different from the last visible
character before it, the ellipsis is displayed with the 'default'
face.

You can read the discussion which led to the current implementation
here:

  http://lists.gnu.org/archive/html/emacs-pretest-bug/2005-04/msg00338.html

Note that the display engine only looks at the face of the first
invisible character; the rest are skipped without looking at their
faces (or any other text properties or overlays).  So if you change
your test case in a very minor way:

  (let ((ov (make-overlay 8 9)))
    (overlay-put ov 'face '(:underline t)))

The "problem" doesn't happen.

> Here is another (more "real-world") way to reproduce this issue (run the
> snippet, then use C-x h in the newly created buffer to mark all text: the
> invisible section isn't highlighted)
> 
> (with-current-buffer (get-buffer-create "org-bug")
>   (erase-buffer)
>   (org-mode)
>   (insert "* line1\n** line2\n* line3")
>   (org-shifttab)
>   (pop-to-buffer (current-buffer)))

You say that this use case is only broken in Emacs 25, but I see this
in all the previous versions as well (as expected, given when this was
coded).  So if you really see this working correctly in Emacs 24.4,
then some other factors on your system were at work here.  Maybe you
didn't test this exact test case there?

> Yet another way it to reproduce it is given below
> 
> (with-current-buffer (get-buffer-create "flyspell-bug")
>   (erase-buffer)
>   (text-mode)
>   (add-to-invisibility-spec '(outline . t))
>   (insert "line1\nline2\nline3")
>   (flyspell-check-region-doublons (point-min) (point-max))
>   (let ((ov (make-overlay 7 12)))
>     (overlay-put ov 'invisible 'outline))
>   (pop-to-buffer (current-buffer)))

What exactly is the problem here?  I don't see anything that looks
problematic.  In particular, there are no duplicate misspelling in
this buffer, so flyspell-check-region-doublons should do nothing.
What am I missing?

> Interestingly, the first example is broken in both 24.4 and emacs-25; but the 
> second one is broken only in emacs-25 (the third one does not run in Emacs 
> 24.4).

This behavior should be present in any Emacs released after Apr 2005.

> I wonder if this is connected to another issue that I've seen, where having a 
> composition (using prettify-symbols-mode) just before an invisible region would 
> cause the background of the invisible region to be incorrect, but setting 
> prettify-symbols-unprettify-at-point and moving the point to the edge of the 
> symbol would cause the background to be fine. Unfortunately I don't have a good 
> repro for that issue.

Hard to tell without a reproducible test case.