bug#18285: 24.3.92; A combination of `display' on text and `invisible' and `before/after-string' leads to the before/after string being displayed twice

[Eli Zaretskii <eliz@gnu.org>]

> From: Dmitry <dgutov@yandex.ru>
> Date: Mon, 18 Aug 2014 02:35:06 +0400
> 
> (let ((pt (point)))
>   (insert (propertize "a" 'display "bbb"))
>   (let ((o (make-overlay pt (point))))
>     (overlay-put o 'invisible t)
>     (overlay-put o 'after-string "foo\nbar")))
> 
> The result looks like this:
> 
> foo
> barbbbfoo
> bar

Believe it or not, but this is how the code was written 14 years ago,
although perhaps this particular consequence wasn't intended.  But it
follows logically from how the display engine works.

The fundamental issue here is that the 'invisible' property makes all
the character positions between the start and the end of the overlay
indistinguishable.  Therefore, the display engine considers the
'before-string' and 'after-string' of an overlay that spans invisible
text to be applicable to both the start and the end of the overlay,
something it wouldn't do if the 'invisible' property were not present.

So what happens is this:

 . we find the 'display' property and arrange for it to be displayed

 . before displaying the 'display' property string, we check for
   overlay strings, and find the 'after-string' whose overlay starts
   at the same position; since its overlay has the 'invisible'
   property, we consider the 'after-string' in effect for the start of
   the overlay, and display that string

 . we display the 'display' string

 . we are going to continue display of the buffer starting at the
   position where the 'display' property ends

 . we check for overlays at that position, and again find the same
   'after-string' because its overlay _ends_ at that buffer position,
   so we display that string again

So you are actually shooting yourself in the foot by using the
'invisible' property here.  Doing that is redundant here, because the
'display' property already makes the text it "covers" invisible.

I think it's not worth to try to fix this (by complicating the heck
out of the display engine) on the account of this use case.

If you have other similar use cases, where simply refraining from
using 'invisible' won't help, please show them.

> It's the same if I use `before-string' instead of `after-string', but
> not `display'.
> 
> If I don't set the `invisible' property, it's just
> 
> bbbfoo
> bar

My crystal ball says you get

 bbbfoo
 bar

only with 'after-string'.  With 'before-string', you should get

 foo
 barbbb

instead.  IOW, without the 'invisible' property getting in the way,
the 'before-string' is displayed before the display property, and
the 'after-string' is displayed after it.

Btw, there was a discussion of a similar issue starting at

  http://lists.gnu.org/archive/html/emacs-devel/2011-04/msg00674.html

It was inconclusive.