bug#19307: 24.4.51; Ellipsis created with `invisible' removes highlighting from overlay after-string after it

bug#19307: 24.4.51; Ellipsis created with `invisible' removes highlighting from overlay after-string after it

[Dmitry Gutov]

In the example below, the `after-string' value is propertized with a
face.

But as long as there is ellipsis at its beginning, the after-string is
rendered using the default face.

(defun test ()
  (interactive)
  (ignore-errors
    (kill-buffer "test"))
  (pop-to-buffer "test")
  (add-to-invisibility-spec '(... . t))
  (insert (propertize "foo" 'invisible '...))
  (let ((ov (make-overlay (point) (point))))
    (overlay-put ov 'invisible t)
    (overlay-put ov 'window (selected-window))
    (overlay-put ov 'after-string
                 (propertize "xxx" 'face 'highlight))))

And here's a somewhat related scenario, with a surprising result:

(defun testt ()
  (interactive)
  (ignore-errors
    (kill-buffer "testt"))
  (pop-to-buffer "testt")
  (add-to-invisibility-spec '(... . t))
  (insert " ")
  (let ((ov (make-overlay (1- (point)) (point))))
    (overlay-put ov 'invisible t)
    (overlay-put ov 'window (selected-window))
    (overlay-put ov 'after-string
                 (propertize "xxx" 'face 'highlight)))
  (insert (propertize "foo" 'invisible '...)))

If I modify the scenario to make the overlay empty (and maybe omit
inserting the space at the beginning, though this makes no difference),
then "xxx" is displayed and even highlighted as expected.

In GNU Emacs 24.4.51.2 (x86_64-unknown-linux-gnu, GTK+ Version 3.10.8)
 of 2014-11-28 on axl
Repository revision: 6b765b8facbdbb03f28028007885236601652515
Windowing system distributor `The X.Org Foundation', version 11.0.11501000
System Description:	Ubuntu 14.04.1 LTS

bug#19307: Acknowledgement (24.4.51; Ellipsis created with `invisible' removes highlighting from overlay after-string after it)

[Dmitry Gutov]

Eli,

Could you look into this?

Thanks.

bug#19307: 24.4.51; Ellipsis created with `invisible' removes highlighting from overlay after-string after it

[Eli Zaretskii]

> Date: Mon, 02 Feb 2015 05:47:11 +0200
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> Eli,
>
> Could you look into this?

Sorry, I somehow missed your original report.

> In the example below, the `after-string' value is propertized with a
> face.
> 
> But as long as there is ellipsis at its beginning, the after-string is
> rendered using the default face.
> 
> (defun test ()
>   (interactive)
>   (ignore-errors
>     (kill-buffer "test"))
>   (pop-to-buffer "test")
>   (add-to-invisibility-spec '(... . t))
>   (insert (propertize "foo" 'invisible '...))
>   (let ((ov (make-overlay (point) (point))))
>     (overlay-put ov 'invisible t)
>     (overlay-put ov 'window (selected-window))
>     (overlay-put ov 'after-string
>                  (propertize "xxx" 'face 'highlight))))

This bug was introduced in Emacs 23, 10 years(!) ago.  Now fixed in
commit 27e11c0 on the emacs-24 branch.

> And here's a somewhat related scenario, with a surprising result:
> 
> (defun testt ()
>   (interactive)
>   (ignore-errors
>     (kill-buffer "testt"))
>   (pop-to-buffer "testt")
>   (add-to-invisibility-spec '(... . t))
>   (insert " ")
>   (let ((ov (make-overlay (1- (point)) (point))))
>     (overlay-put ov 'invisible t)
>     (overlay-put ov 'window (selected-window))
>     (overlay-put ov 'after-string
>                  (propertize "xxx" 'face 'highlight)))
>   (insert (propertize "foo" 'invisible '...)))

This is unrelated, AFAICT, and is not a bug: what you have here is 2
chunks of invisible text, one after another.  The display engine skips
all of that, and never examines any additional properties or overlays
in the middle of the invisible text.  Emacs always worked like that.

> If I modify the scenario to make the overlay empty (and maybe omit
> inserting the space at the beginning, though this makes no difference),
> then "xxx" is displayed and even highlighted as expected.

Each one of the measures you describe either removes one of the
invisible chunks of text or makes it visible.  That's why the overlay
string becomes displayed then.  (As for it being highlighted, the
above bug affected the highlight only when the overlay string
_follows_ the ellipsis; if it comes before the ellipsis, the bug won't
rear its ugly head.)

Is there some important real-life use case that bumped into this
surprise?  If so, please describe it.

Thanks.

bug#19307: 24.4.51; Ellipsis created with `invisible' removes highlighting from overlay after-string after it

[Dmitry Gutov]

On 02/02/2015 06:29 PM, Eli Zaretskii wrote:

> This bug was introduced in Emacs 23, 10 years(!) ago.

Don't know your feelings on this subject, but I'm kinda proud. :)

 > Now fixed in
> commit 27e11c0 on the emacs-24 branch.

Thanks, but it introduced a regression. Insert some text at the end, and 
"xxx" will get displayed twice:

(defun test2 ()
   (interactive)
   (ignore-errors
     (kill-buffer "test"))
   (pop-to-buffer "test")
   (add-to-invisibility-spec '(... . t))
   (insert (propertize "foo" 'invisible '...))
   (let ((ov (make-overlay (point) (point))))
     (overlay-put ov 'invisible t)
     (overlay-put ov 'window (selected-window))
     (overlay-put ov 'after-string
                  (propertize "xxx" 'face 'highlight)))
   (insert " "))

>> And here's a somewhat related scenario, with a surprising result:
>>
...
> This is unrelated, AFAICT, and is not a bug: what you have here is 2
> chunks of invisible text, one after another.  The display engine skips
> all of that, and never examines any additional properties or overlays
> in the middle of the invisible text.  Emacs always worked like that.

I see. Well, that unfortunate. I can only say that it goes against my 
expectations.

> Is there some important real-life use case that bumped into this
> surprise?  If so, please describe it.

Not into the second one, so far. But the first example caused the 
Company tooltip lose color when displayed after an outline.

Not to diminish your efforts, but I've noticed that the fix for each 
display problem I've reported lately involved move added lines than 
removed ones. Which looks like adding more special cases. That's worrying.

bug#19307: 24.4.51; Ellipsis created with `invisible' removes highlighting from overlay after-string after it

[Eli Zaretskii]

> Date: Tue, 03 Feb 2015 05:33:02 +0200
> From: Dmitry Gutov <dgutov@yandex.ru>
> CC: 19307@debbugs.gnu.org
> 
>     This bug was introduced in Emacs 23, 10 years(!) ago.
> 
> Don't know your feelings on this subject, but I'm kinda proud. :)

You should be.

> > Now fixed in
> 
>     commit 27e11c0 on the emacs-24 branch.
> 
> Thanks, but it introduced a regression. Insert some text at the end, and "xxx" will get displayed twice:

Turns out the bug I fixed concealed another one that was also there
for a long time (more than 9 years), and became exposed due to my fix.
Now fixed in commit e589765 on the emacs-24 branch.

> Not to diminish your efforts, but I've noticed that the fix for each display problem I've reported lately involved move added lines than removed ones. Which looks like adding more special cases. That's worrying.

The bugs happened in special cases for which no one coded the
solution, so catering to those cases often needs additional code.
Isn't that natural?

Say you have something like

  int reckless_division = a / b;

and then someone reports a divide-by-zero crash, and you add
protection against b being zero -- won't you expect the code to grow a
little?

Anyway, you should be happier with this last fix, since it removes
more lines than it adds ;-)

bug#19307: 24.4.51; Ellipsis created with `invisible' removes highlighting from overlay after-string after it

[Dmitry Gutov]

On 02/03/2015 08:54 PM, Eli Zaretskii wrote:

> Turns out the bug I fixed concealed another one that was also there
> for a long time (more than 9 years), and became exposed due to my fix.
> Now fixed in commit e589765 on the emacs-24 branch.

Terrific. Works fine, AFAICS.

> The bugs happened in special cases for which no one coded the
> solution, so catering to those cases often needs additional code.
> Isn't that natural?

Okay. As long as you're confident that the cases are sufficiently 
special, I'm going to be content, too.

> Anyway, you should be happier with this last fix, since it removes
> more lines than it adds ;-)

Thanks, I am. :-)