bug#22320: Overlays with an 'invisible property break stacking of overlay faces

[Eli Zaretskii <eliz@gnu.org>]

> Cc: 22320@debbugs.gnu.org
> From: Clément Pit--Claudel <clement.pitclaudel@live.com>
> Date: Thu, 7 Jan 2016 12:03:38 -0500
> 
> >   http://lists.gnu.org/archive/html/emacs-pretest-bug/2005-04/msg00338.html
> 
> Interesting, thanks for the pointer. I read the discussion, but I see no rationale there for this choice. Was one ever given?

I do see a rationale provided here:

  http://lists.gnu.org/archive/html/emacs-pretest-bug/2005-04/msg00339.html

The change causes the code to keep using the same face in cases where
that is not controversial, and fall back on 'default' otherwise.

> Why is this better than, for example, keeping the face of the first invisible character and extending it to the whole ellipsis?

That would go back to the original problem, whereby each ellipsis
could have a different face depending on the faces in the invisible
text.  I think it's confusing to have the faces of the invisible text
show through, don't you agree?  Ellipsis just indicates there's some
hidden text, why should it "inherit" the face of that hidden text?

> Am I mistaken to think that this choice is also inconsistent with the way faces on composed characters are handled? It seems that composing a sequence of characters into "..." will keep the face of the first compose character. That default is useful in many places (such as Arthur's nameless mode, or in flyspell or flycheck when an error covers a sequence of characters composed into a single one). Is there a good reason to treat ellipses standing for invisible spans differently?

I'm not sure I understand what you describe here.  Can you show me
some simple Lisp which demonstrates the situation?

> The interactions of the current behaviour with transient mark mode are especially confusing

I agree, but I learned long ago that one should use invisible text as
little as possible, and in particular expect that text properties and
overlays put on invisible text will have some specific effect.  The
display engine skips the invisible text entirely, looking only at its
first character (and sometimes the last); if you think about this, you
will immediately understand that having properties and overlays on
invisible text is playing with fire -- basically, you bump into
undefined behavior.

> Using transient-mark-mode and selecting characters one by one from the end of the buffer, "i", "h", and "g" are progressively selected, then nothing happens despite "def" being in fact selected (so pressing C-w at that point kills more than highlighted), then "def" and "c" get highlighted at the same time. This means that the highlighted region doesn't actually correspond to the marked region. Is that also expected?

The ellipsis gets highlighted as soon as the last visible character
before the invisible text has the same face as the invisible text.
That is what you see.  So yes, this is expected.

Like I said: it's a hard problem, and the solution only caters to the
simple, no-brainer, use cases.  I'm not surprised that you can come up
with weird situations, but I cannot see a better, more general
solution here.

> >> (with-current-buffer (get-buffer-create "org-bug")
> >>   (erase-buffer)
> >>   (org-mode)
> >>   (insert "* line1\n** line2\n* line3")
> >>   (org-shifttab)
> >>   (pop-to-buffer (current-buffer)))
> > 
> > You say that this use case is only broken in Emacs 25, but I see this
> > in all the previous versions as well (as expected, given when this was
> > coded).  So if you really see this working correctly in Emacs 24.4,
> > then some other factors on your system were at work here.  Maybe you
> > didn't test this exact test case there?
> 
> You're right; this example works fine in 24.3, but it is indeed broken in 24.5. I tested both with -Q:

No, it behaves the same in 24.3 and in 24.5 (and in all versions
before that).  I don't understand how you see something different.

>    Works:  GNU Emacs 24.3.1 (x86_64-unknown-linux-gnu, GTK+ Version 3.10.8) of 2015-06-17 on clem-w50-mint
>    Broken: GNU Emacs 24.5.2 (x86_64-unknown-linux-gnu, GTK+ Version 3.10.8) of 2015-06-20 on clem-w50-mint
> 
> Is this something that changed in org then?

No, nothing changed.  Which version of Org did you use in each Emacs
release?  (I used the one bundled with that release.)

> >> (with-current-buffer (get-buffer-create "flyspell-bug")
> >>   (erase-buffer)
> >>   (text-mode)
> >>   (add-to-invisibility-spec '(outline . t))
> >>   (insert "line1\nline2\nline3")
> >>   (flyspell-check-region-doublons (point-min) (point-max))
> >>   (let ((ov (make-overlay 7 12)))
> >>     (overlay-put ov 'invisible 'outline))
> >>   (pop-to-buffer (current-buffer)))
> > 
> > What exactly is the problem here?  I don't see anything that looks
> > problematic.  In particular, there are no duplicate misspelling in
> > this buffer, so flyspell-check-region-doublons should do nothing.
> > What am I missing?
> 
> Indeed, I use aspell instead of ispell. It seems to ignore numbers, and thus flags the first two instances of "line" as duplicates.

And what problems do you see then?