bug#37829: 27.0.50; Overlay behaviour changed without documentation.

bug#37829: 27.0.50; Overlay behaviour changed without documentation.

[Zhu Zihao]

Reproduce step:

1. open a empty buffer, type "Lorem ipsum" in it, then goto point-min
2. Eval this code

(let ((ov (make-overlay (point-min) (1+ (point-at-eol)))))
  (overlay-put ov 'face 'mode-line))

In current version of Emacs, the mode line overlay will only cover the string
"Lorem ipsum", but in 26.2, the overlay will cover the hole line(every pixel in
line).

It may be a bug because no documentation or NEWS mentioned this change.

In GNU Emacs 27.0.50 (build 11, x86_64-pc-linux-gnu, GTK+ Version 3.24.12, cairo version 1.17.3)
 of 2019-10-20 built on archlinux
Repository revision: cea9577b7d6fcf01599afd48078f8ff1defb1297
Repository branch: makepkg
Windowing system distributor 'The X.Org Foundation', version 11.0.12005000
System Description: Arch Linux

bug#37829: 27.0.50; Overlay behaviour changed without documentation.

[Eli Zaretskii]

> Date: Sun, 20 Oct 2019 16:49:10 +0800
> From: Zhu Zihao <all_but_last@163.com>
> 
> 1. open a empty buffer, type "Lorem ipsum" in it, then goto point-min
> 2. Eval this code
> 
> (let ((ov (make-overlay (point-min) (1+ (point-at-eol)))))
>   (overlay-put ov 'face 'mode-line))
> 
> In current version of Emacs, the mode line overlay will only cover the string
> "Lorem ipsum", but in 26.2, the overlay will cover the hole line(every pixel in
> line).
> 
> It may be a bug because no documentation or NEWS mentioned this change.

Crystal ball says it's because the mode-line face doesn't have the
:extend attribute by default.  If so, this change _is_ in NEWS and in
the ELisp manual.

Thanks.

bug#37829: 27.0.50; Overlay behaviour changed without documentation.

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Crystal ball says it's because the mode-line face doesn't have the
> :extend attribute by default.  If so, this change _is_ in NEWS and in
> the ELisp manual.

Hasn't there been sufficient fallout now from this change that we should
consider doing the :extend stuff the opposite way?  That is, default to
the old behaviour, and if we want certain faces not to extend, then we
put in :extend nil on those faces.

This change breaks a lot of out-of-tree code.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#37829: 27.0.50; Overlay behaviour changed without documentation.

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Zhu Zihao <all_but_last@163.com>,  37829@debbugs.gnu.org
> Date: Sun, 20 Oct 2019 13:16:08 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Crystal ball says it's because the mode-line face doesn't have the
> > :extend attribute by default.  If so, this change _is_ in NEWS and in
> > the ELisp manual.
> 
> Hasn't there been sufficient fallout now from this change that we should
> consider doing the :extend stuff the opposite way?

No, I don't think so.  All the complaints until now were about Diff
mode and related modes, and we made the relevant faces extend a few
hours ago.  At least in one case, this change allowed to _remove_ some
kludgey code (in info.el).

And doing this the opposite way makes no sense to me, it's in effect
the same as removing the feature.

> This change breaks a lot of out-of-tree code.

No, it doesn't break any code.  It changes how display looks in some
cases, so people are surprised at first.

For this particular bug report, why would someone expect the overlay's
color to extend to the end of the line instead of affecting only the
text that the overlay covers?

bug#37829: 27.0.50; Overlay behaviour changed without documentation.

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> And doing this the opposite way makes no sense to me, it's in effect
> the same as removing the feature.

I don't understand.  Doing it the opposite way would be just as
expressive.

> > This change breaks a lot of out-of-tree code.
>
> No, it doesn't break any code.  It changes how display looks in some
> cases, so people are surprised at first.

You may quibble, but changing the look this radically is breaking the
code for me.

> For this particular bug report, why would someone expect the overlay's
> color to extend to the end of the line instead of affecting only the
> text that the overlay covers?

Because that's the way Emacs has worked since forever: If you put a face
on a newline, then it'll extend to the end of the line.

We do that because that's the way we wanted the display to look.  If we
didn't want that, we didn't put the face on the newline.  

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#37829: 27.0.50; Overlay behaviour changed without documentation.

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: all_but_last@163.com,  37829@debbugs.gnu.org
> Date: Sun, 20 Oct 2019 13:43:58 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > And doing this the opposite way makes no sense to me, it's in effect
> > the same as removing the feature.
> 
> I don't understand.  Doing it the opposite way would be just as
> expressive.

The idea behind this feature was that most faces shall not be
extended, so doing it the opposite way would mean we need to change
the definitions of an unlimited number of faces, including those not
in core.

> > > This change breaks a lot of out-of-tree code.
> >
> > No, it doesn't break any code.  It changes how display looks in some
> > cases, so people are surprised at first.
> 
> You may quibble, but changing the look this radically is breaking the
> code for me.

I suggest to run with it for some time, you may change your mind.  It
happened to many of us.

> > For this particular bug report, why would someone expect the overlay's
> > color to extend to the end of the line instead of affecting only the
> > text that the overlay covers?
> 
> Because that's the way Emacs has worked since forever: If you put a face
> on a newline, then it'll extend to the end of the line.

We considered that a side effect of the implementation.  Many other
applications don't do that.

> We do that because that's the way we wanted the display to look.  If we
> didn't want that, we didn't put the face on the newline.  

Others said the exact opposite: that they want to be able to do that
without having the face extended.  Also, the automatic extension in
Emacs 26 and before behaved inconsistently in GUI and text-mode
frames, and even between different attributes (color vs underline, for
example).

bug#37829: 27.0.50; Overlay behaviour changed without documentation.

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> The idea behind this feature was that most faces shall not be
> extended, so doing it the opposite way would mean we need to change
> the definitions of an unlimited number of faces, including those not
> in core.

We do not have to change anything not in core -- whether people want the
new, more convenient behaviour, is up to them.

And there certainly aren't unlimited places we have to change thing
in-tree, because most things in-tree look just how we wanted them to.

> I suggest to run with it for some time, you may change your mind.  It
> happened to many of us.

I know that the new interface is convenient -- I'll be able to ditch a
bunch of code in shr that works around the problem.  But that's just
it -- this is what everybody has done forever, and have ended up with
code that does exactly what they want it to.  (I.e., placing a face on
newline to extend the face to the end of the line.)

>> We do that because that's the way we wanted the display to look.  If we
>> didn't want that, we didn't put the face on the newline.  
>
> Others said the exact opposite: that they want to be able to do that
> without having the face extended.

With the new interface, they can do that, whatever the default is.  The
question is whether Emacs should do this massive, extremely user-visible
(with very ugly results) thing by default.

I think no.

> Also, the automatic extension in Emacs 26 and before behaved
> inconsistently in GUI and text-mode frames, and even between different
> attributes (color vs underline, for example).

Well, the only attributes where it makes a difference are background
colours and underline, surely?  (Well, and overline, but nobody uses
that.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#37829: 27.0.50; Overlay behaviour changed without documentation.

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: all_but_last@163.com,  37829@debbugs.gnu.org
> Date: Sun, 20 Oct 2019 18:49:23 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > The idea behind this feature was that most faces shall not be
> > extended, so doing it the opposite way would mean we need to change
> > the definitions of an unlimited number of faces, including those not
> > in core.
> 
> We do not have to change anything not in core -- whether people want the
> new, more convenient behaviour, is up to them.

But under the assumption that most faces should not be extended, that
would mean our default is wrong in most cases, and what kind of
default is that?

> And there certainly aren't unlimited places we have to change thing
> in-tree, because most things in-tree look just how we wanted them to.

I said "including those not in core".

> > Others said the exact opposite: that they want to be able to do that
> > without having the face extended.
> 
> With the new interface, they can do that, whatever the default is.

They wanted to do that without customizing the faces.

> > Also, the automatic extension in Emacs 26 and before behaved
> > inconsistently in GUI and text-mode frames, and even between different
> > attributes (color vs underline, for example).
> 
> Well, the only attributes where it makes a difference are background
> colours and underline, surely?  (Well, and overline, but nobody uses
> that.)

No, there's also strikethrough and box.

bug#37829: 27.0.50; Overlay behaviour changed without documentation.

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> But under the assumption that most faces should not be extended, that
> would mean our default is wrong in most cases, and what kind of
> default is that?

I agree that it's a bad default, but changing it now is going to require
that a lot of people is going to have to change their code.

>> And there certainly aren't unlimited places we have to change thing
>> in-tree, because most things in-tree look just how we wanted them to.
>
> I said "including those not in core".

We don't have to change things not in core.

I'm for not changing defaults like this unnecessarily.  And it is
unnecessary -- things work for people now (because they've adapted to
the behaviour), and having the default be the established behaviour
wouldn't limit the usability of the feature.

But I'm going to stop arguing about this now, because we're just
restating our arguments.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#37829: 27.0.50; Overlay behaviour changed without documentation.

[Zhu Zihao]

On Sun, 20 Oct 2019 19:05:26 +0800,
Eli Zaretskii wrote:
> 
> > Date: Sun, 20 Oct 2019 16:49:10 +0800
> > From: Zhu Zihao <all_but_last@163.com>
> > 
> > 1. open a empty buffer, type "Lorem ipsum" in it, then goto point-min
> > 2. Eval this code
> > 
> > (let ((ov (make-overlay (point-min) (1+ (point-at-eol)))))
> >   (overlay-put ov 'face 'mode-line))
> > 
> > In current version of Emacs, the mode line overlay will only cover the string
> > "Lorem ipsum", but in 26.2, the overlay will cover the hole line(every pixel in
> > line).
> > 
> > It may be a bug because no documentation or NEWS mentioned this change.
> 
> Crystal ball says it's because the mode-line face doesn't have the
> :extend attribute by default.  If so, this change _is_ in NEWS and in
> the ELisp manual.
> 
> Thanks.

Thanks, I just search "overlay" in NEWS so I didn't notice it's the issue of
face, not the overlay.

bug#37829: 27.0.50; Overlay behaviour changed without documentation.

[Zhu Zihao]

I'm puzzled that we already have :extend, but when I set the end of an overlay
to the EOL, the face of overlay still don't cover the whole line.

I have to set it to the BOL of next line.

If I understand correctly, :extend already mark a face extends EOL, why we still
need old method to make it take effect?

bug#37829: 27.0.50; Overlay behaviour changed without documentation.

[Eli Zaretskii]

> Date: Mon, 21 Oct 2019 00:14:18 +0800
> From: Zhu Zihao <all_but_last@163.com>
> Cc: Zhu Zihao <all_but_last@163.com>,
> 	37829@debbugs.gnu.org
> 
> 
> I'm puzzled that we already have :extend, but when I set the end of an overlay
> to the EOL, the face of overlay still don't cover the whole line.

The position you use as the end of the overlay is one position after
the end of the text that is affected by the overlay.  Remember: in
Emacs, a buffer position is between two characters.

bug#37829: 27.0.50; Overlay behaviour changed without documentation.

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> Crystal ball says it's because the mode-line face doesn't have the
> :extend attribute by default.  If so, this change _is_ in NEWS and in
> the ELisp manual.

OK; doesn't seem to be a bug here, so I'm closing this bug report.  If
there's still something to fix here, please reply to the debbugs
address, and we'll reopen the report.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no