bug#15899: 24.3.50; regression: `region' overlay is lower priority than default

bug#15899: 24.3.50; regression: `region' overlay is lower priority than default

[Drew Adams]

In Emacs 24.3, if you highlight some text with an overlay, without
specifying any priority for the overlay, that overlay will be lower
priority than the `region' face overlay that is used to select text for
the region.

In this build, the region overlay is of lower priority, so the other
overlay hides the region overlay.

To test, create an overlay without specifying any priority.
E.g. evaluate this in buffer *scratch* from emacs -Q.

(text-mode) ; No font lock interference.
(setq overlay  (make-overlay 20 40))
(overlay-put overlay 'face 'highlight)

Then select text in the first line, from before char 20 to after char
40.  The highlighting from 20 to 40 shows, instead of the region.

Do the same thing in Emacs 24.3 - no bug there.  Dunno when the
regression started.


In GNU Emacs 24.3.50.1 (i686-pc-mingw32)
 of 2013-11-12 on LEG570
Windowing system distributor `Microsoft Corp.', version 6.1.7601
Configured using:
 `configure --enable-checking 'CFLAGS=-O0 -g3' CPPFLAGS=-DGLYPH_DEBUG=1'

bug#15899: 24.3.50; regression: `region' overlay is lower priority than default

[Eli Zaretskii]

> Date: Thu, 14 Nov 2013 14:57:14 -0800 (PST)
> From: Drew Adams <drew.adams@oracle.com>
> 
> In Emacs 24.3, if you highlight some text with an overlay, without
> specifying any priority for the overlay, that overlay will be lower
> priority than the `region' face overlay that is used to select text for
> the region.
> 
> In this build, the region overlay is of lower priority, so the other
> overlay hides the region overlay.

In Emacs 24.3, the region highlight was not done by an overlay, but by
special code in C.  Now it is an overlay, so the issue of priority
creeps in.

> Do the same thing in Emacs 24.3 - no bug there.  Dunno when the
> regression started.

It started when Stefan made the region use an overlay.

bug#15899: 24.3.50; regression: `region' overlay is lower priority than default

[Stefan Monnier]

> In Emacs 24.3, the region highlight was not done by an overlay, but by
> special code in C.  Now it is an overlay, so the issue of priority
> creeps in.

Indeed.  OTOH, I'm not convinced it's a bug, since this "bug" was also
the fix for another bug.  It's a change, there's no doubt about that.
Whether it's better or worse is not so clear.


        Stefan

bug#15899: 24.3.50; regression: `region' overlay is lower priority than default

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Drew Adams <drew.adams@oracle.com>,  15899@debbugs.gnu.org
> Date: Fri, 15 Nov 2013 08:54:15 -0500
> 
> I'm not convinced it's a bug, since this "bug" was also the fix for
> another bug.

Was that other bug also about priorities of faces?

> It's a change, there's no doubt about that.  Whether it's better or
> worse is not so clear.

Isn't it confusing that the region highlighting is non-contiguous when
an overlay is in its middle?

What are the downsides of setting the region overlay's priority to
most-positive-fixnum?

bug#15899: 24.3.50; regression: `region' overlay is lower priority than default

[Dmitry Gutov]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Cc: Drew Adams <drew.adams@oracle.com>,  15899@debbugs.gnu.org
>> Date: Fri, 15 Nov 2013 08:54:15 -0500
>> 
>> I'm not convinced it's a bug, since this "bug" was also the fix for
>> another bug.
>
> Was that other bug also about priorities of faces?

Yes. http://debbugs.gnu.org/cgi/bugreport.cgi?bug=15618

bug#15899: 24.3.50; regression: `region' overlay is lower priority than default

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,  15899@debbugs.gnu.org
> Date: Fri, 15 Nov 2013 17:32:07 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Stefan Monnier <monnier@iro.umontreal.ca>
> >> Cc: Drew Adams <drew.adams@oracle.com>,  15899@debbugs.gnu.org
> >> Date: Fri, 15 Nov 2013 08:54:15 -0500
> >> 
> >> I'm not convinced it's a bug, since this "bug" was also the fix for
> >> another bug.
> >
> > Was that other bug also about priorities of faces?
> 
> Yes. http://debbugs.gnu.org/cgi/bugreport.cgi?bug=15618

But since the region is now an overlay, the inconsistency will be
gone, right?

bug#15899: 24.3.50; regression: `region' overlay is lower priority than default

[Dmitry Gutov]

On 15.11.2013 18:40, Eli Zaretskii wrote:
>>>> I'm not convinced it's a bug, since this "bug" was also the fix for
>>>> another bug.
>>>
>>> Was that other bug also about priorities of faces?
>>
>> Yes. http://debbugs.gnu.org/cgi/bugreport.cgi?bug=15618
>
> But since the region is now an overlay, the inconsistency will be
> gone, right?

Yes, but if the region overlay will have priority infinity, the 
inconsistency will "be gone" in the opposite way from how 15618 was 
resolved.

Which will make the related feature of `easy-kill' much harder (maybe 
impossible) to implement.

If the region overlay will have a high but finite and documented 
priority, that would be much better.

bug#15899: 24.3.50; regression: `region' overlay is lower priority than default

[Eli Zaretskii]

> Date: Sat, 16 Nov 2013 00:35:25 +0200
> From: Dmitry Gutov <dgutov@yandex.ru>
> CC: monnier@iro.umontreal.ca, 15899@debbugs.gnu.org
> 
> On 15.11.2013 18:40, Eli Zaretskii wrote:
> >>>> I'm not convinced it's a bug, since this "bug" was also the fix for
> >>>> another bug.
> >>>
> >>> Was that other bug also about priorities of faces?
> >>
> >> Yes. http://debbugs.gnu.org/cgi/bugreport.cgi?bug=15618
> >
> > But since the region is now an overlay, the inconsistency will be
> > gone, right?
> 
> Yes, but if the region overlay will have priority infinity, the 
> inconsistency will "be gone" in the opposite way from how 15618 was 
> resolved.

But the behavior will still be consistent.  And complaint about
inconsistency is how I read that bug report.  It even asks "which is
the right behavior?", implying that having it consistent either way
would be OK.

> Which will make the related feature of `easy-kill' much harder (maybe 
> impossible) to implement.

Can you tell more about this feature, and why it cares to be "more
equal" than the region?  (Sorry, I don't have time to read the source
or try it.)  Why is it important for easy-kill overlay to make region
highlighting invisible?

> If the region overlay will have a high but finite and documented 
> priority, that would be much better.

Which will start an "overlay priority arms race", something I loathe.

bug#15899: 24.3.50; regression: `region' overlay is lower priority than default

[Jarek Czekalski]

W dniu 2013-11-16 09:49, Eli Zaretskii pisze:
>> If the region overlay will have a high but finite and documented
>> priority, that would be much better.
> Which will start an "overlay priority arms race", something I loathe.

 From this point of view editors like Notepad are best. No races, no 
possibility of user or package interfering with application author's 
vision. But we are in Emacs. This should mean freedom to users. If a 
user wants to have a higher priority, why would you forbid him to do so?

I can't imagine an example where infinite priority is better than a high 
value. Could you help with that? I guess examples of malicious users or 
those who don't read docs should not count.

Jarek

bug#15899: 24.3.50; regression: `region' overlay is lower priority than default

[Eli Zaretskii]

> Date: Sat, 16 Nov 2013 10:51:55 +0100
> From: Jarek Czekalski <jarekczek@poczta.onet.pl>
> 
> W dniu 2013-11-16 09:49, Eli Zaretskii pisze:
> >> If the region overlay will have a high but finite and documented
> >> priority, that would be much better.
> > Which will start an "overlay priority arms race", something I loathe.
> 
>  From this point of view editors like Notepad are best. No races, no 
> possibility of user or package interfering with application author's 
> vision. But we are in Emacs. This should mean freedom to users. If a 
> user wants to have a higher priority, why would you forbid him to do so?

Every freedom must have its limits.  "Your freedom to swing fists ends
where my nose begins."  (Yes, I know I'm lecturing, but so did you.)

More to the point: Previously, Emacs users did not have the freedom to
overrule the region highlighting with an overlay face.  Many
generations of Emacs users lived with that limitation and never
complained about that, at least not seriously enough to make this an
issue.  Keeping the priority of the region overlay at infinity just
preserves previous behavior.

So I think we should turn the table and ask why would a user need to
have this freedom now, and only give that freedom if the cause
justifies it.

> I can't imagine an example where infinite priority is better than a high 
> value. Could you help with that?

It avoids the problem of priority race.  With an infinite priority, we
can be sure the region highlighting will always be visible, come what
may.

> I guess examples of malicious users or those who don't read docs
> should not count.

No, but unintended consequences of actions by unsuspecting users
should.  In a complex system, unintended consequences are always a
greater danger than malicious intent.

IOW, keeping the region priority above everything makes sure we won't
have another series of bug reports in the near future asking why this
or that feature makes region invisible.

bug#15899: 24.3.50; regression: `region' overlay is lower priority than default

[Jarek Czekalski]

W dniu 2013-11-16 11:42, Eli Zaretskii pisze:
> Every freedom must have its limits. "Your freedom to swing fists ends 
> where my nose begins." (Yes, I know I'm lecturing, but so did you.)

Sorry if you didn't like my tone. I tried to make a general point and 
justify it well. On the other hand, I don't mind lecturing like yours, 
if that's what we call lecturing here.

> So I think we should turn the table and ask why would a user need to
> have this freedom now, and only give that freedom if the cause
> justifies it.

Good point. I would do it just for the sake of flexibility, which Emacs 
should be proud of. We may not predict in what way people will want to 
user overlays. And some of them may be silently disappointed if 
flexibility of overlays is not sufficient. They even won't complain 
about it, so we may never hear such a request. But if flexibility is 
achieved, there may be silent happy users. That's a benefit.

Introducing a new feature needs considering pros and cons. Personally I 
don't see enough cons. Dmitri's answer presents ways to deal with 
potential problems. Good documentation would be the most important 
weapon. Something like: "It is strongly suggested not to specify a 
priority higher than ..., because it will cause problems with displaying 
selection boundaries."

An example of such need just came to my mind. A temporary overlay which 
highlights for a second the words spoken by a user through a microphone.

So if I were to decide, I would say: freedom and flexibility!

Sorry again for lecturing. And being pathetic too :)

Jarek

bug#15899: 24.3.50; regression: `region' overlay is lower priority than default

[Drew Adams]

> Previously, Emacs users did not have the freedom to
> overrule the region highlighting with an overlay face.  Many
> generations of Emacs users lived with that limitation and never
> complained about that, at least not seriously enough to make this an
> issue.  Keeping the priority of the region overlay at infinity just
> preserves previous behavior.

No, it does not preserve previous behavior.

1. Previously, isearch highlighting took priority over region
highlighting.  Isearch highlighting has a large, finite value.

2. Previously, region highlighting used the `face' text property.
3rd-party code and user code and interactions that manipulate text
properties will no longer work.

Yes, using an overlay for the region brings other possibilities.
But it takes away certain possibilities, as well (#2).

> > I can't imagine an example where infinite priority is better than
> > a high value. Could you help with that?
> 
> It avoids the problem of priority race.

If you declare a winner ahead of time then there is no race.  Big
deal.  Isearch needs (by default, at least) to win out over region.
And why not others as well, depending on need?  Why foreclose the
possibility?

Sure, if someone wants to fiddle with priorites then there is always
the potential for escalation spiraling round.  That's life with Lisp.

> With an infinite priority, we can be sure the region highlighting
> will always be visible, come what may.

Which might not always be what someone wants.

I'm all for region appearing on top, by default, as I've already
argued in detail.  Users need to know what they've selected.  But
that does not mean that someone should be prevented from doing things
otherwise.

> IOW, keeping the region priority above everything makes sure we
> won't have another series of bug reports in the near future asking
> why this or that feature makes region invisible.

That's just an argument for the default behavior.  "_Keeping_ the
priority above everything else" is wrong.  _Setting_ it above everything
else (with exceptions such as isearch) by default is right.

bug#15899: 24.3.50; regression: `region' overlay is lower priority than default

[Eli Zaretskii]

> Date: Sat, 16 Nov 2013 08:23:35 -0800 (PST)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: 15899@debbugs.gnu.org
> 
> 2. Previously, region highlighting used the `face' text property.

No, region never used any text properties.  There was a 'region' face,
yes, but no text property.  Instead, the display engine had special
code for displaying the active region in the 'region' face.

bug#15899: 24.3.50; regression: `region' overlay is lower priority than default

[Stefan Monnier]

> 2. Previously, region highlighting used the `face' text property.
> 3rd-party code and user code and interactions that manipulate text
> properties will no longer work.

Obviously you don't know what you're talking about.


        Stfean

bug#15899: 24.3.50; regression: `region' overlay is lower priority than default

[Drew Adams]

> > 2. Previously, region highlighting used the `face' text property.
> > 3rd-party code and user code and interactions that manipulate text
> > properties will no longer work.
> 
> Obviously you don't know what you're talking about.

Feel better now, Stefan?

I already said several hours ago, after Eli pointed it out, that
I was mistaken in thinking that face `region' was applied as a
text property.  I was (mistakenly) convinced that I had seen it
listed in `C-u C-x =' output as such.  Mea culpa (again).

At least I know enough to guess that selection highlighting
should, at the very least by default, visibly cover the entire
selection.  That's a no-brainer, in my book.  But then, I don't
know what I'm talking about.

bug#15899: 24.3.50; regression: `region' overlay is lower priority than default

[Dmitry Gutov]

On 16.11.2013 10:49, Eli Zaretskii wrote:
>> Yes, but if the region overlay will have priority infinity, the
>> inconsistency will "be gone" in the opposite way from how 15618 was
>> resolved.
>
> But the behavior will still be consistent.  And complaint about
> inconsistency is how I read that bug report.  It even asks "which is
> the right behavior?", implying that having it consistent either way
> would be OK.

The bug report was what is was, but `easy-kill' depends on region 
highlighting working in a certain way.


>> Which will make the related feature of `easy-kill' much harder (maybe
>> impossible) to implement.
>
> Can you tell more about this feature, and why it cares to be "more
> equal" than the region?  (Sorry, I don't have time to read the source
> or try it.)  Why is it important for easy-kill overlay to make region
> highlighting invisible?

It has a command `easy-mark' which selects some unit of text around 
point. And it uses a dedicated overlay to mark the place where point was 
before the command was called, in color.

So that overlay needs to have higher priority than region. No need to 
make region highlighting invisible.

>> If the region overlay will have a high but finite and documented
>> priority, that would be much better.
>
> Which will start an "overlay priority arms race", something I loathe.

I don't think so. The region overlay priority won't change, even if 
people decide to shoot themselves in the foot and raise priorities of 
overlays inappropriately.

bug#15899: 24.3.50; regression: `region' overlay is lower priority than default

[Eli Zaretskii]

> Date: Sat, 16 Nov 2013 12:25:36 +0200
> From: Dmitry Gutov <dgutov@yandex.ru>
> CC: monnier@iro.umontreal.ca, 15899@debbugs.gnu.org
> 
> `easy-kill' depends on region highlighting working in a certain way.

What way is that?  Before region highlighting was reimplemented as an
overlay, it had a fixed "priority" that couldn't be controlled or
worked around.  So what exactly did easy-kill expect from that
behavior?

> It has a command `easy-mark' which selects some unit of text around 
> point. And it uses a dedicated overlay to mark the place where point was 
> before the command was called, in color.

But the region always has point on one of its ends, so both the places
where point was and where it is are clearly visible when the region is
active.  So why is there a need for the easy-mark to be visible in
that situation (which is transient and therefore short-lived)?

> So that overlay needs to have higher priority than region. No need to 
> make region highlighting invisible.

The part of the region that overlaps the easy-mark overlay will not be
visible, if the region's priority is lower.  Or did you mean something
else?

> >> If the region overlay will have a high but finite and documented
> >> priority, that would be much better.
> >
> > Which will start an "overlay priority arms race", something I loathe.
> 
> I don't think so. The region overlay priority won't change, even if 
> people decide to shoot themselves in the foot and raise priorities of 
> overlays inappropriately.

That's not the race I had in mind.  What I had in mind was users
complaining about their favorite overlay-based features being obscured
by the region, lobbying the maintainers to increase the priorities of
those overlays above the region (and possibly also above the easy-mark
overlay), or increase the priority of the region overlay; then other
users complaining about the effects of that, and so on and so forth ad
nauseam.  How can you even assume that the "region overlay priority
won't change", given the possibility and enough pressure from users?
Once out of the bottle, this genie cannot be easily put back.

bug#15899: 24.3.50; regression: `region' overlay is lower priority than default

[Dmitry Gutov]

On 16.11.2013 13:24, Eli Zaretskii wrote:
> What way is that?  Before region highlighting was reimplemented as an
> overlay, it had a fixed "priority" that couldn't be controlled or
> worked around.  So what exactly did easy-kill expect from that
> behavior?

That behavior was broken on Leo Liu's system (don't know how) and on 
mine too (with some themes only).

See https://github.com/leoliu/easy-kill/issues/3 for more details.

> But the region always has point on one of its ends, so both the places
> where point was and where it is are clearly visible when the region is
> active.

In many times point wasn't at any of region ends. That's how `easy-mark' 
works, it selects some unit of text *around* the point.

> So why is there a need for the easy-mark to be visible in
> that situation (which is transient and therefore short-lived)?

Just for user information. Looks kinda nice.

> The part of the region that overlaps the easy-mark overlay will not be
> visible, if the region's priority is lower.

That just means that easy-mark will need to set its priority higher than 
the *documented* region priority.

> That's not the race I had in mind.  What I had in mind was users
> complaining about their favorite overlay-based features being obscured
> by the region, lobbying the maintainers to increase the priorities of
> those overlays above the region (and possibly also above the easy-mark
> overlay), or increase the priority of the region overlay; then other
> users complaining about the effects of that, and so on and so forth ad
> nauseam.

This will have to be solved on a package-by-package basis, like every 
contentious feature. One user complaint: change the priority of a 
package-specific overlay. More complaints to change it in different 
directions: add a user option that will set the priority that overlay.

 > How can you even assume that the "region overlay priority
> won't change", given the possibility and enough pressure from users?
> Once out of the bottle, this genie cannot be easily put back.

Region priority will have to be set once (to 100, or something), 
documented, and then never changed again. Any changes will have to be 
made to other overlays' priorities.

bug#15899: 24.3.50; regression: `region' overlay is lower priority than default

[Drew Adams]

> > ... users complaining about the effects of that, and so on and
> > so forth ad nauseam.
> 
> This will have to be solved on a package-by-package basis, like
> every contentious feature. One user complaint: change the priority
> of a package-specific overlay. More complaints to change it in
> different directions: add a user option that will set the priority
> that overlay.

Yes.

> > How can you even assume that the "region overlay priority
> > won't change", given the possibility and enough pressure from
> > users? Once out of the bottle, this genie cannot be easily put back.
> 
> Region priority will have to be set once (to 100, or something),
> documented, and then never changed again. Any changes will have to
> be made to other overlays' priorities.

Yes.  But users will also be able to change the priority of the
region overlay.

You are both right: The overlay genie is a genie, and it will
definitely be out of the bottle (Eli).  It is only the default
region overlay that needs to be defined (well).  Users changing
that or other priorities can do good and or harm.  Emacs Dev
need not change the default priority of the region (or any other)
overlay, unless it thinks that is appropriate (Dmitry).

Think isearch overlay priorities.  Have they ever changed?  Has
there been a deluge of user whining that they need to be changed?

bug#15899: 24.3.50; regression: `region' overlay is lower priority than default

[Drew Adams]

> It has a command `easy-mark' which selects some unit of text around
> point. And it uses a dedicated overlay to mark the place where point
> was before the command was called, in color.
> 
> So that overlay needs to have higher priority than region. No need
> to make region highlighting invisible.

Similarly, the isearch overlays should have higher priority than any
region overlay.

(And yes, I agree with you that users should of course be able to
change priorities as they see fit.)

If region highlighting is to use an overlay, then its default priority
should be higher than most predefined (i.e., emacs -Q) overlays, but
lower than the isearch priorities.

bug#15899: 24.3.50; regression: `region' overlay is lower priority than default

[Drew Adams]

> > I don't think so. The region overlay priority won't change, even
> > if people decide to shoot themselves in the foot and raise
> > priorities of overlays inappropriately.
> 
> That's not the race I had in mind.  What I had in mind was users
> complaining about their favorite overlay-based features being
> obscured by the region, lobbying the maintainers to increase the
> priorities of those overlays above the region (and possibly also
> above the easy-mark overlay), or increase the priority of the region
> overlay; then other users complaining about the effects of that,
> and so on and so forth ad nauseam.  How can you even assume that
> the "region overlay priority won't change", given the possibility
> and enough pressure from users?

User complaint escalation is the escalation you fear?  That's a new
one.  Emacs Dev has never had much trouble resisting user complaints
to change defaults.  And foreclosing such debate is never the
solution anyway.

Just set the default priority for the region overlay to a value close
to but lower than the isearch overlays.  If there are future
suggestions to tweak default priorities then that will be discussed
concretely when it comes up.  If users - or you - have good reasons
to change default priorities, then those can be adjusted.

> Once out of the bottle, this genie cannot be easily put back.

Once face `region' is used for an overlay and not a text property,
the overlay genie is out of the bottle, yes.  New possibilities
arrive, and some old possibilities leave.

bug#15899: 24.3.50; regression: `region' overlay is lower priority than default

[Stefan Monnier]

> Isn't it confusing that the region highlighting is non-contiguous when
> an overlay is in its middle?

1- you need more than "an overlay in its middle": you need this overlay
   to put a face property that happens to completely cancel the region's
   own face properties (since the `face' properties of overlapping
   overlays are merged).
2- I don't think it's particularly confusing, no.  Usually the context
   makes it pretty clear, and if the user is surprised at some point,
   that surprise will most likely not last very long.
So, no, I don't really think it's a bug.

> What are the downsides of setting the region overlay's priority to
> most-positive-fixnum?

I most-positive-fixnum-ly hate overlay priorities.


        Stefan

bug#15899: 24.3.50; regression: `region' overlay is lower priority than default

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: drew.adams@oracle.com,  15899@debbugs.gnu.org
> Date: Fri, 15 Nov 2013 20:25:46 -0500
> 
> > Isn't it confusing that the region highlighting is non-contiguous when
> > an overlay is in its middle?
> 
> 1- you need more than "an overlay in its middle": you need this overlay
>    to put a face property that happens to completely cancel the region's
>    own face properties (since the `face' properties of overlapping
>    overlays are merged).

It's enough for that face to specify a background color, no?  That's
not uncommon for Emacs features.  E.g., try "M-x hl-line-mode RET".

> 2- I don't think it's particularly confusing, no.  Usually the context
>    makes it pretty clear, and if the user is surprised at some point,
>    that surprise will most likely not last very long.

Well, a few people just disagreed with you.

> > What are the downsides of setting the region overlay's priority to
> > most-positive-fixnum?
> 
> I most-positive-fixnum-ly hate overlay priorities.

No offense, but I think we can live with that downside ;-)
Are there any others?

In any case, the moment you reimplemented the region as an overlay,
you got us this issue, because it is inherent in the use of overlays,
and cannot be escaped.

bug#15899: 24.3.50; regression: `region' overlay is lower priority than default

[Stefan Monnier]

>> > Isn't it confusing that the region highlighting is non-contiguous when
>> > an overlay is in its middle?
>> 1- you need more than "an overlay in its middle": you need this overlay
>> to put a face property that happens to completely cancel the region's
>> own face properties (since the `face' properties of overlapping
>> overlays are merged).
> It's enough for that face to specify a background color, no?

In some cases, yes, because the region's foreground color is
often unnoticeable (e.g. same as default).

>> I most-positive-fixnum-ly hate overlay priorities.
> No offense, but I think we can live with that downside ;-)

The downside is not that I hate it, but the reasons why I hate it: it's
as much a source of problems as a solution.  `priorities' impose a total
ordering, where often there isn't one: in some circumstance one overlay
should be on top, in others it's the other way around.

The "default priority" at least is able to handle those things
sometimes, by making overlays's ordering depending on nesting.

> In any case, the moment you reimplemented the region as an overlay,
> you got us this issue, because it is inherent in the use of overlays,
> and cannot be escaped.

It was present before as well.  The behavior was different but was also
a source of "priority problems".

My intuition tells me that if Emacs had use the current system for the
last 20 years and had just changed to the "region is always at the very
top", people would complain just as much.


        Stefan

bug#15899: 24.3.50; regression: `region' overlay is lower priority than default

[Drew Adams]

> > It's enough for that face to specify a background color, no?
> 
> In some cases, yes, because the region's foreground color is
> often unnoticeable (e.g. same as default).

That's only by default.  We should not base too many assumptions
on any particular face attributes for face `region'.  It is user
customizable.  Better to plan for it to be like any other face:
any attributes at all, at least as much as we can.

> >> I most-positive-fixnum-ly hate overlay priorities.
> > No offense, but I think we can live with that downside ;-)
> 
> The downside is not that I hate it, but the reasons why I hate
> it: it's as much a source of problems as a solution.
> `priorities' impose a total ordering, where often there isn't
> one: in some circumstance one overlay should be on top, in
> others it's the other way around.

I don't disagree (and I'm glad that you are giving reasons ;-)).
But what is a better approach?  A total ordering is black & white,
but at least it gives people a degree of control.  And at least
that control is simple: a total ordering is a simple model.

How about giving an example of a problem?  And a solution -
something that solves that problem and gives users more (not
less) control.

> The "default priority" at least is able to handle those things
> sometimes, by making overlays's ordering depending on nesting.

Not sure what that means, and I wish I did understand what you
mean by that.  Can you give a tiny example to illustrate?

> > In any case, the moment you reimplemented the region as an
> > overlay, you got us this issue, because it is inherent in
> > the use of overlays, and cannot be escaped.
> 
> It was present before as well.  The behavior was different
> but was also a source of "priority problems".

Not clear how so.  Can you elaborate?  Are you referring to
the fact that a user who wants to see some other highlighting
(besides isearch) "on top" could not do so?  That I can see.
If you mean something else then I don't know what it is.

> My intuition tells me that if Emacs had use the current system
> for the last 20 years and had just changed to the "region is
> always at the very top", people would complain just as much.

People sometimes complain less when (a) the new behavior is
proposed and explained and (b) they have an opportunity to
question and discuss it.

We are certainly doing that here, now, but this is something
that would be more appropriate for emacs-devel, IMO.  It would
have been better to initiate a discussion and proposal there,
pointing to the bug report and outlining what the behavior
changes would be.  But you've heard this before...

bug#15899: 24.3.50; regression: `region' overlay is lower priority than default

[Stefan Monnier]

>> The "default priority" at least is able to handle those things
>> sometimes, by making overlays's ordering depending on nesting.
> Not sure what that means, and I wish I did understand what you
> mean by that.  Can you give a tiny example to illustrate?

The default ordering between two overlays that have the same priority is
that if one of the two overlays is nested in the other it should have
higher priority.  The idea is to try and avoid having one overlay
completely hide another.

> Not clear how so.  Can you elaborate?  Are you referring to
> the fact that a user who wants to see some other highlighting
> (besides isearch) "on top" could not do so?

Exactly.

> We are certainly doing that here, now, but this is something
> that would be more appropriate for emacs-devel, IMO.  It would
> have been better to initiate a discussion and proposal there,
> pointing to the bug report and outlining what the behavior
> changes would be.  But you've heard this before...

No, the change's purpose was not to fix that bug.  It was just
a side-effect.  The main purpose of the change was to move the region
highlighting code out of the redisplay to Elisp.

The good news is that if you liked to old behavior you can get it now by
writing some Elisp code, whereas in the past you had no such choice.


        Stefan

bug#15899: 24.3.50; regression: `region' overlay is lower priority than default

[Daniel Colascione]

On 11/16/2013 09:45 AM, Stefan Monnier wrote:
>>>> Isn't it confusing that the region highlighting is non-contiguous when
>>>> an overlay is in its middle?
>>> 1- you need more than "an overlay in its middle": you need this overlay
>>> to put a face property that happens to completely cancel the region's
>>> own face properties (since the `face' properties of overlapping
>>> overlays are merged).
>> It's enough for that face to specify a background color, no?
>
> In some cases, yes, because the region's foreground color is
> often unnoticeable (e.g. same as default).
>
>>> I most-positive-fixnum-ly hate overlay priorities.
>> No offense, but I think we can live with that downside ;-)
>
> The downside is not that I hate it, but the reasons why I hate it: it's
> as much a source of problems as a solution.  `priorities' impose a total
> ordering, where often there isn't one: in some circumstance one overlay
> should be on top, in others it's the other way around.

Can you provide an example of an actual case where two overlays should 
be ordered one way in one context and another in a different context? 
Nothing comes to mind at the moment.

I don't think numeric priorities are as big a social problem as you 
suspect: in other cases where we use priorities to manage interaction of 
unrelated modifiers on some base behavior (e.g., NT filter driver 
altitudes), priorities work well enough and don't lead to arms races. 
I'd have preferred an ordered list to numeric priorities, but numeric 
priorities are good enough.

bug#15899: 24.3.50; regression: `region' overlay is lower priority than default

[Stefan Monnier]

> Can you provide an example of an actual case where two overlays should be
> ordered one way in one context and another in a different context? Nothing
> comes to mind at the moment.

No, I admit to not having a concrete case to give you offhand (I do
remember that we have bumped into such problems in the past, tho).

The issue boils down to the fact that we generally don't want an overlay
to hide another.  But priorities are not a good way to solve this
problem, since the "hiding" depends on the relative position: if an
overlay A is nested within another overlay B, then A should be "on top"
in order not to be hidden, and if later the relative position of
A changes such that B is now nested into A, then B should now be "on top".

Sometimes we really do want one overlay to "be on top", including hiding
another, and in that case priorities can work OK.

> I don't think numeric priorities are as big a social problem as you suspect:

Indeed, there's also the general problems of priorities, e.g. where
A wants to be on top of B, C wants to be on top of A and B wants to be
on top of C.  And I agree that in practice these issues aren't all that
bad.  But overlays have their own additional issues, specific to
their nature.


        Stefan

bug#15899: 24.3.50; regression: `region' overlay is lower priority than default

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Isn't it confusing that the region highlighting is non-contiguous when
>> an overlay is in its middle?
>
> 1- you need more than "an overlay in its middle": you need this overlay
>    to put a face property that happens to completely cancel the region's
>    own face properties (since the `face' properties of overlapping
>    overlays are merged).
> 2- I don't think it's particularly confusing, no.  Usually the context
>    makes it pretty clear, and if the user is surprised at some point,
>    that surprise will most likely not last very long.
> So, no, I don't really think it's a bug.

Closing.

-- 
(domestic pets only, the antidote for overdose, milk.)
  bloggy blog http://lars.ingebrigtsen.no/

bug#15899: 24.3.50; regression: `region' overlay is lower priority than default

[Drew Adams]

> > In Emacs 24.3, the region highlight was not done by an overlay,
> > but by special code in C.  Now it is an overlay, so the issue
> > of priority creeps in.
> 
> Indeed.  OTOH, I'm not convinced it's a bug, since this "bug" was
> also the fix for another bug.

Which other bug?  Was that bug there ever since the Emacs region
could be highlighted (i.e., several decades old)?  Or was that
bug introduced recently, as the result of some other change?
Maybe there is more than one change that needs to be reverted?

Oh, I see that Dmitry has now answered that question: 
http://debbugs.gnu.org/cgi/bugreport.cgi?bug=15618

Which was NOT a bug, IMO.  It should not have been "fixed".  The
fact that face `region' highlights "on top of" other highlighting
is _on purpose_, i.e., by design.  Countering that is folly, IMO.

What next?  Someone wants to see "theme" highlighting on top of
Isearch overlays?  And someone eagerly jumps in to "fix" that
"bug" too?

> It's a change, there's no doubt about that. Whether it's better
> or worse is not so clear.

Are you saying that you think it might not be a bug (regression)
to use an overlay rather than a text property?  That would be a
reasonable question.

Or are you suggesting that it might not be a bug that selecting
text does not show the selection face throughout the selection?

The latter would be unreasonable.  When you select text you
should always be able to see which text you selected - all of it,
even if it is only one character.  This should be a no-brainer.
The whole point of having a `region' face is to highlight what
is in the region.

(FWIW, I'm not sure I'm in favor of the change to using an
overlay from using a text property either, but that is not what
this bug report is about.  Using the region to select text, and
then replacing face `region' text property with another face or
another text property, or otherwise making use of the property,
can be useful.  Sure, you can make do without `region' as a text
property, but why should you have to?  Why did this behavior
suddenly need to be changed, after four decades or so of use?)

Please:

1. If you keep the use of an overlay, at *least* give users a
choice (e.g. a user option) of whether face `region' is to be
applied as a text property or used for an overlay.

2. When it is used for an overlay, please make its priority
higher than others, with the exception of isearch overlays
(yes, there can sometimes be a use for continuing to show the
region while isearching).  If there are any other exceptions
needed to bring the behavior back to pre-regression, those
would apply also.  (I can't think of any other exceptions,
right now).

These kinds of changes should first be the subject of proposal
and discussion in emacs-devel, and perhaps a user poll.  They
should not be made willy nilly when fixing a bug.  IMHO.

bug#15899: 24.3.50; regression: `region' overlay is lower priority than default

[Stefan Monnier]

> is _on purpose_, i.e., by design.

No, it was the result of the implementation technique, no of design.


        Stefan

bug#15899: 24.3.50; regression: `region' overlay is lower priority than default

[Drew Adams]

> > The fact that face `region' highlights "on top of" other
> > highlighting is _on purpose_, i.e., by design.
> 
> No, it was the result of the implementation technique, no
> of design.

You "accidentally" cut off that first line, where I said
_what_ was by design.  I've put it back, to restore the
message honestly.

How do you know that Emacs Dev did not _intend_ for the
region display to appear "on top of" other highlighting
(with the exception of isearch)?

I don't believe for a second that this was by accident or
just a result of an "implementation technique".  Not until
you can show some supporting evidence for that claim.

Without contrary evidence, I am persuaded that Emacs, like
every other app that uses selection, chose this behavior for
its _user-level_ effect: The point is to show users just what
is selected.  There is no better reason to make such a choice.  
Result of an "implementation technique", indeed!

And there were already lots of other UIs that showed
selection highlighting, before Emacs added its own.  Just as
for menus, tooltips, mouse-face highlighting, links, text
attributes, and much of the rest of the Emacs UI, these things
were developed first outside Emacs and only later (sometimes
much later) emulated by and added to Emacs.  Emacs got its
selection highlighting design from others, not as a result
of some Emacs "implementation technique".

Users everywhere expect a selection highlight to show them
clearly what they've selected.  That, and not some unspoken
"implementation technique", is the reason that Emacs Dev
showed the region face clearly throughout the region selected.
Up until this regression, that is.

If what is now the behavior remains, then Emacs will be the
only UI I'm aware of that does not have selection
highlighting override other highlighting in appearance - the
only one where _you cannot know what you've selected_ based
on highlighting.  Can you name another UI that does that?

This is really, really silly.  Usability out the window - poof.

Users will have no idea what they are selecting.  Except in
lucky cases.  And even then they will have to remain unsure,
because there is no way to tell whether what they are seeing
highlighted as selected is really everything that is selected
or just some of it.

> Usually the context makes it pretty clear,

"Usually" there is no other highlighting of the same text.
So what?  This usual case is not what the bug is about.

> and if the user is surprised at some point, that surprise
> will most likely not last very long

Wrong.  They will need to remain unsure - there is no way
to know whether something unhighlighted with face `region' is
in fact selected.

And even if you were right that user confusion would "most
likely not last very long" (based on what?), there is _no
reason_ that users have to be confused about selection
coverage at all.  This-won't-hurt-long is no consolation,
because there is no need to inflict pain in the first place.

No reason not to show them the region highlighted normally.

bug#15899: 24.3.50; regression: `region' overlay is lower priority than default

[Barry OReilly]

[-- Attachment #1: Type: text/plain, Size: 494 bytes --]

I agree having the region overlay at lowered priority is undesirable.
I think most other overlays are longer lived, and during the shorter
periods I use the region, it's the one I'm most interested in seeing.

> Isn't it confusing that the region highlighting is non-contiguous
> when an overlay is in its middle?

Actually, it's worse when the other overlay is at either end of the
active region, because then I can't tell for sure where the region
began or ended unless I happen to remember.

[-- Attachment #2: Type: text/html, Size: 581 bytes --]

bug#15899: 24.3.50; regression: `region' overlay is lower priority than default

[Drew Adams]

> > 2. Previously, region highlighting used the `face' text property.
> 
> No, region never used any text properties.  There was a 'region'
> face, yes, but no text property.  Instead, the display engine had
> special code for displaying the active region in the 'region' face.

You are right about that, Eli.  I was quite mistaken about it.

Funny (embarrassing), I could have sworn that I had seen face
`region' listed as a text property in `C-u C-x =', but I must not
have.  And yes, I should have double-checked that.

Thank you for the correction.  And that changes what I said about
moving `region' to an overlay taking some text-property behavior
away from Lisp users.

With that information, I now see no downside (so far) to moving
face `region' to an overlay.  We just need to get the overlay
(by default) to appear to be on top of most other overlays (with
isearch as an exception).

bug#15899: 24.3.50; regression: `region' overlay is lower priority than default

[Drew Adams]

> > 1. I meant that the change to the behavior of the region not
> >    appearing "on top of" other highlighting (except isearch)
> >    needs to be reverted (undone).
> 
> You cannot revert behavior, only the code.

I added "(undone)", to be clear.  And no, code is not the only
thing that can be reverted.  When you revert a buffer using
`g', you are reverting behavior/effects, not reverting code.

Revert just means to go back to a previous state.

> If the new implementation has unwanted side effects, those side
> effects need to be fixed by further changes.
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

No, that is not the need.  The need is to remove the unwanted side
effects - sure.  But that does not imply keeping the culprit code
and then trying to make "further changes" to make things right.

That is only one possible approach.  It is not _necessary_ to
follow that approach -there is no such "need".

Another approach is to (1) revert the code changes that brought
about the regression and then (2) fix the bug those changes were
intended to fix in some other way.

That is, assuming there really was a bug to be fixed.  Can you
say what it was?  If so, can you also say why keeping the changes
made so far and applying (unknown) "further changes" is the best
way to go?

> > 4. If it was in fact a bug, it's not clear why the fix for it
> > needed to involve changing region highlighting to use an
> > overlay.  Not clear to me anyway.
> 
> It doesn't have to be clear.

I assume you mean that it does not have to be clear to me.
Hopefully you agree that it has to be clear to someone.

Is it clear to you?  If so, do tell, pray.

> The fact that region highlighting now uses an overlay is an
> implementation detail.

No, it is not.  Why?  Because Emacs _users_ do things with buffer
text, and with text properties, and with overlays.  These are not
simply implementation artifacts.  They are first class Emacs
thingies that can be and are manipulated by Emacs users.

Some users might not care whether some particular highlighting is
via a text property or an overlay.  Other users might care.
Depends on what they are doing.  Users interact with Emacs in
lots of ways besides interactive use of `emacs -Q' commands and
menus.  They add text properties, examine them, change them,
search their text, change their text,...

This is a passably big change for Emacs Users.  Emacs has used
text property `face' to "implement" region highlighting ever
since such highlighting has existed.  Why no proposal for such
a change?  No discussion?

> Bug reports should generally remain on the level of behavior,
> i.e. requirements, they should not normally go to the
> implementation level.

What is the _behavioral requirement_ that obliges the use of an
overlay for the region?  Can you answer that?  I've seen nothing
so far.

Talk about text properties vs overlays is on the user level, as
well as the developer level.  It is mistaken to think that it is
something hidden to users inside some "implementation level".

> The implementors should have freedom to implement the
> required behavior as they see fit, as long as the results are
> reasonable.

Sure they should.  But just what is the required behavior here?
And how relative is that "freedom"?  And what is the community
of "implementors" that should enjoy this freedom?

"The implementors" are not a separate breed from Emacs users.
We are in this together.

("The educator himself must be educated." - Theses on Feuerbach)

> > My suggestion is to first revert the code change and then
> > discuss what the bug is that it was intended to fix.  If
> > there is really a bug that needs fixing, then let's please
> > try to find some other, non-shotgun fix for it.
> 
> Again, please stay on the level of required behavior, and
> leave the implementation out of this discussion.

Something wrong with discussing implementation among
"implementors"?  Among users too?  Not at all.

> As long as there's no evidence that the new implementation
                     ^^^^^^^^^^^
> cannot possibly accommodate the required behavior, the
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^
> implementation can stay.

Setting the bar a bit high, aren't you?

Balderdash.  As long as there is no evidence that the _old_
implementation "cannot possibly accommodate" other changes that
fix the bug, the old implementation can stay.

You are being simplistic here.  The new implementation is not
the One Tried & True that cannot be questioned.  It is not
arguments against the new implementation that need to be proved
beyond a reasonable doubt.  That's backwards.

What should be demonstrated is why we should choose a new
region implementation.  This change has not even been released
yet - why on Earth put up a wall of "as long as there is no
evidence that it cannot possibly accommodate"?  As if it were
enshrined in unquestionable tradition.

It's bad enough to defend a longstanding and proven status quo
dogmatically.  It's downright ridiculous to do so for a change
that is only a few days old and has not yet seen the Light of
User Day.

Is there even any "evidence" that there was a bug to be fixed?
Again, can you say what the bug is, clearly and not referring
to implementation?

IOW:

  This is the _requirement_, in terms of behavior: _____

  And this is why it is better to fix the remaining problems
  caused by the recent "fix" than it is to start over with a
  different fix: _______

bug#15899: 24.3.50; regression: `region' overlay is lower priority than default

[Drew Adams]

> > This change should be reverted as soon as possible.
> 
> It needs not be reverted, I think.  We just need to make the
> priority of the region overlay higher than any other overlay.

1. I meant that the change to the behavior of the region not
appearing "on top of" other highlighting (except isearch) needs
to be reverted (undone).  IOW, bug #15899 is a regression.  It
is the behavior that needs to be fixed, regardless of how that
is done.

2. Region highlighting should *not* be higher priority than
all other overlays.  It should not be higher than isearch
highlighting, for instance.  There might be other exceptions
too; dunno.  See my previous mail.

3. It's not clear that the reported "bug" was in fact a bug,
rather than the intended Emacs behavior.  Or if it was, it is
not clear just what the bug was.  If it was thought to be a
bug that other highlighting was in some cases overruled
by region highlighting, then that was not, IMO, a bug.

4. If it was in fact a bug, it's not clear why the fix for it
needed to involve changing region highlighting to use an
overlay.  Not clear to me anyway.

My suggestion is to first revert the code change and then
discuss what the bug is that it was intended to fix.  If
there is really a bug that needs fixing, then let's please
try to find some other, non-shotgun fix for it.

There should be no need to change the longstanding behavior
of the Emacs region just because someone's highlighting does
not show through.

bug#15899: 24.3.50; regression: `region' overlay is lower priority than default

[Eli Zaretskii]

> Date: Fri, 15 Nov 2013 09:14:37 -0800 (PST)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: monnier@iro.umontreal.ca, 15899@debbugs.gnu.org
> 
> > > This change should be reverted as soon as possible.
> > 
> > It needs not be reverted, I think.  We just need to make the
> > priority of the region overlay higher than any other overlay.
> 
> 1. I meant that the change to the behavior of the region not
> appearing "on top of" other highlighting (except isearch) needs
> to be reverted (undone).

You cannot revert behavior, only the code.  If the new implementation
has unwanted side effects, those side effects need to be fixed by
further changes.

> 2. Region highlighting should *not* be higher priority than
> all other overlays.  It should not be higher than isearch
> highlighting, for instance.  There might be other exceptions
> too; dunno.  See my previous mail.

I don't disagree.  If there are other overlays that should show
through the region, they should have higher priority.

> 4. If it was in fact a bug, it's not clear why the fix for it
> needed to involve changing region highlighting to use an
> overlay.  Not clear to me anyway.

It doesn't have to be clear.  The fact that region highlighting now
uses an overlay is an implementation detail.  Bug reports should
generally remain on the level of behavior, i.e. requirements, they
should not normally go to the implementation level.  The implementors
should have freedom to implement the required behavior as they see
fit, as long as the results are reasonable.

> My suggestion is to first revert the code change and then
> discuss what the bug is that it was intended to fix.  If
> there is really a bug that needs fixing, then let's please
> try to find some other, non-shotgun fix for it.

Again, please stay on the level of required behavior, and leave the
implementation out of this discussion.  As long as there's no evidence
that the new implementation cannot possibly accommodate the required
behavior, the implementation can stay.

> There should be no need to change the longstanding behavior
> of the Emacs region just because someone's highlighting does
> not show through.

Agreed.

bug#15899: 24.3.50; regression: `region' overlay is lower priority than default

[Drew Adams]

> Isn't it confusing that the region highlighting is non-contiguous
> when an overlay is in its middle?

Yes.  And when a `face' or other display-related text property is
in its middle.  Or is at either end.  Or extends across the whole
region (which includes when the region is in the middle of a
stretch of highlighting).  I know from experience with the new
(and old) behavior.

Such situations are visually confusing to users.  And they can
complicate operating on text properties and, more generally,
using text properties and overlays.

This change should be reverted as soon as possible.  It is a
candidate for "What WERE they thinking?"  At the very least,
make such behavior optional for users (and certainly not the
default).

There are thousands of real bugs reported, many of them still
unresponded to.  There is no reason to spend time "fixing"
non-bugs such as this.  This should have gotten only a polite
"This is by design", perhaps followed by an explanation of why
the design (in Emacs and everywhere else!) is thus.

bug#15899: 24.3.50; regression: `region' overlay is lower priority than default

[Eli Zaretskii]

> Date: Fri, 15 Nov 2013 07:55:27 -0800 (PST)
> From: Drew Adams <drew.adams@oracle.com>
> Cc: 15899@debbugs.gnu.org
> 
> > Isn't it confusing that the region highlighting is non-contiguous
> > when an overlay is in its middle?
> 
> Yes.  And when a `face' or other display-related text property is
> in its middle.  Or is at either end.  Or extends across the whole
> region (which includes when the region is in the middle of a
> stretch of highlighting).  I know from experience with the new
> (and old) behavior.
> 
> Such situations are visually confusing to users.  And they can
> complicate operating on text properties and, more generally,
> using text properties and overlays.
> 
> This change should be reverted as soon as possible.

It needs not be reverted, I think.  We just need to make the priority
of the region overlay higher than any other overlay.