bug#42347: Feature request: Visual block attribute for overlays

bug#42347: Feature request: Visual block attribute for overlays

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

In Emacs 21 to 26, overlays between two points in the buffer on two 
different lines extend to the right border of the window.  This has 
changed in Emacs 27, and now the default is that overlays extend only one 
character position after the last character of the line.  The previous 
behavior can be obtained with the ":extend t" face attribute.  I agree 
that the earlier behavior was not optimal, but I think the current 
behavior with its staircase aspect is not optimal either.

I think a third way to display overlays would make sense, and would be 
better than the earlier and current defaults.  Let's name this attribute 
":visualblock".  It would produce the following result:

1. calculate the overlay it it would have been displayed by Emacs 21-26.

2. remove all pixel columns on the right *and on the left* of the overlay 
which have no "content" (that is, no characters on the right, and 
whitespace characters on the left).

With this, on overlay on, for example, a block of code between () or {} 
would be displayed on the screen as a block.  The above "algorithm" works 
with fixed and variable-width fonts, but could be made more efficient for 
fixed-width fonts.

Gregory

bug#42347: Feature request: Visual block attribute for overlays

[Eli Zaretskii]

> Date: Wed, 8 Jul 2020 19:19:05 +0200 (CEST)
> From: Gregory Heytings via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> 1. calculate the overlay it it would have been displayed by Emacs 21-26.
> 
> 2. remove all pixel columns on the right *and on the left* of the overlay 
> which have no "content" (that is, no characters on the right, and 
> whitespace characters on the left).

What would be the definition of "whitespace" for this purpose?

bug#42347: Feature request: Visual block attribute for overlays

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

This should be merged with bug #42307.  (It was sent earlier, and for some 
reason stayed in the mail queue during a week.)

>
>> 1. calculate the overlay it it would have been displayed by Emacs 
>> 21-26.
>>
>> 2. remove all pixel columns on the right *and on the left* of the 
>> overlay which have no "content" (that is, no characters on the right, 
>> and whitespace characters on the left).
>
> What would be the definition of "whitespace" for this purpose?
>

I think the regexp "[ \t\r\n]*" would be appropriate.

Gregory

bug#42347: Feature request: Visual block attribute for overlays

[Eli Zaretskii]

> Date: Tue, 14 Jul 2020 07:38:38 +0000
> From: Gregory Heytings via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> >> 2. remove all pixel columns on the right *and on the left* of the 
> >> overlay which have no "content" (that is, no characters on the right, 
> >> and whitespace characters on the left).
> >
> > What would be the definition of "whitespace" for this purpose?
> >
> 
> I think the regexp "[ \t\r\n]*" would be appropriate.

Are you sure?

First, \r doesn't happen in Emacs buffers, and \n is the newline, so
it isn't on the same line.

Next, what about \f, and more generally about any other sequence of
characters that match [:blank:]?

And finally, what about stretches of whitespace generated by the
'space' display properties?

bug#42307: bug#42347: Feature request: Visual block attribute for overlays

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>>>> 2. remove all pixel columns on the right *and on the left* of the 
>>>> overlay which have no "content" (that is, no characters on the right, 
>>>> and whitespace characters on the left).
>>>
>>> What would be the definition of "whitespace" for this purpose?
>>>
>>
>> I think the regexp "[ \t\r\n]*" would be appropriate.
>
> Are you sure?
>
> First, \r doesn't happen in Emacs buffers,
>

I wasn't sure about \r, but included it just in case.

>
> and \n is the newline, so it isn't on the same line.
>

It is, but it's the last character of the line.  With the current default 
behavior it is displayed, there is one more blank character after each 
line, just type C-x = on that character.

In the two pictures (visualblock.png and visualblock-2.png) I sent (see 
bug #42307), the \n is removed on the right.  Otherwise there would be one 
more column on the right.

>
> Next, what about \f, and more generally about any other sequence of 
> characters that match [:blank:]?
>

Good question.  As far as I know \f is displayed by default as '^L' by 
Emacs, so it's a visible character, and I'm fine with this.  I think it 
should not count as a blank character, which means that an overlay with a 
'\f' in a column would extend at least to that column.

>
> And finally, what about stretches of whitespace generated by the 'space' 
> display properties?
>

I don't know, and I'm not sure I fully understand the question.  The 
"algorithm" I proposed is, I think, clear and simple: draw the overlay 
with :extend t, and trim whitespace on the left and on the right.

That being said, on a second thought I think a better name for the 
proposed attribute would be ":trim" (or perhaps ":trimleft" and 
"trimright" to give as much possible freedom to Emacs users).  It could 
have, for example, three values:

- default value = nil, gives the default behavior (with and without 
:extend)

- value = t, gives the behavior of "visiblechars.png"

- value = column, gives the behavior of "visualblock.png" and 
"visualblock-2.png"

Gregory

bug#42307: bug#42347: Feature request: Visual block attribute for overlays

[Eli Zaretskii]

> Date: Tue, 14 Jul 2020 15:45:01 +0000
> From: Gregory Heytings via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> > and \n is the newline, so it isn't on the same line.
> >
> 
> It is, but it's the last character of the line.  With the current default 
> behavior it is displayed, there is one more blank character after each 
> line, just type C-x = on that character.

My point is that \n cannot be "leading space" of a line.  It is the
end of the previous line.

> > And finally, what about stretches of whitespace generated by the 'space' 
> > display properties?
> 
> I don't know, and I'm not sure I fully understand the question.

The point is that the result of displaying these properties is exactly
the same as tabs and spaces.  So excluding them would surprise users.

bug#42307: Feature request: Visual block attribute for overlays

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>>> and \n is the newline, so it isn't on the same line.
>>>
>>
>> It is, but it's the last character of the line.  With the current 
>> default behavior it is displayed, there is one more blank character 
>> after each line, just type C-x = on that character.
>
> My point is that \n cannot be "leading space" of a line.  It is the end 
> of the previous line.
>

Yes, now I see what you mean.  So the regexp for "blank" on the left would 
be "[ \t]*", and the regexp for "blank" on the right would be "\n%*", 
where "%" denotes "no character".

>
>>> And finally, what about stretches of whitespace generated by the 
>>> 'space' display properties?
>>
>> I don't know, and I'm not sure I fully understand the question.
>
> The point is that the result of displaying these properties is exactly 
> the same as tabs and spaces.  So excluding them would surprise users.
>

Okay.  So... let's include them ;-)

In fact, I'm not entirerly sure that explaining (or implementing) that 
behavior with regexpes or character is the best thing to do.  It's a 
matter of visual representation.  Perhaps the following explanation is 
clearer (or more precise): draw the overlay with :extend t, and remove 
pixel columns on the left and on the right that are displayed in the same 
way as a whitespace character would have been displayed.

Gregory

bug#42307: Feature request: Visual block attribute for overlays

[Eli Zaretskii]

> Date: Tue, 14 Jul 2020 17:01:43 +0000
> From: Gregory Heytings via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> In fact, I'm not entirerly sure that explaining (or implementing) that 
> behavior with regexpes or character is the best thing to do.  It's a 
> matter of visual representation.  Perhaps the following explanation is 
> clearer (or more precise): draw the overlay with :extend t, and remove 
> pixel columns on the left and on the right that are displayed in the same 
> way as a whitespace character would have been displayed.

Well, it's not that simple.  For starters, there's no "pixel columns"
in the display engine, at least not on GUI frames.

More to the point, I think the feature you requested will be not very
simple to implement, because it requires the display code to do
look-ahead: when it lays out leading whitespace, it should consider
characters after that.  That calls for some complications, because the
display code basically inspects characters one by one and makes
(almost) all the decisions on the fly.

So someone will have fun implementing this.  Patches welcome.

bug#42307: Feature request: Visual block attribute for overlays

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>
>> In fact, I'm not entirerly sure that explaining (or implementing) that 
>> behavior with regexpes or character is the best thing to do.  It's a 
>> matter of visual representation.  Perhaps the following explanation is 
>> clearer (or more precise): draw the overlay with :extend t, and remove 
>> pixel columns on the left and on the right that are displayed in the 
>> same way as a whitespace character would have been displayed.
>
> Well, it's not that simple.  For starters, there's no "pixel columns" in 
> the display engine, at least not on GUI frames.
>

That's what I thought indeed.  My explanation was a specification, not an 
implementation hint ;-)

>
> More to the point, I think the feature you requested will be not very 
> simple to implement, because it requires the display code to do 
> look-ahead: when it lays out leading whitespace, it should consider 
> characters after that.  That calls for some complications, because the 
> display code basically inspects characters one by one and makes (almost) 
> all the decisions on the fly.
>
> So someone will have fun implementing this.  Patches welcome.
>

I'm not sure I have the necessary background to do this (it's a 
euphemism), but I'll try to see what I can do.

Gregory

bug#42347: Feature request: Visual block attribute for overlays

[Drew Adams]

> The previous behavior can be obtained with the ":extend t" face attribute.

Only on an individual basis, right?  If you want to
get the previous behavior everywhere, do you need
to change every overlay?

Or is there an option for that?  If not, why not?

And is there a way to get the previous behavior by
default, and something that does the opposite of
`:extend' for individual cases where you want the
new behavior?

If not, why not?

bug#42347: Feature request: Visual block attribute for overlays

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>
>> The previous behavior can be obtained with the ":extend t" face 
>> attribute.
>
> Only on an individual basis, right?  If you want to get the previous 
> behavior everywhere, do you need to change every overlay?
>

AFAIK, yes.

>
> Or is there an option for that?  If not, why not?
>
> And is there a way to get the previous behavior by default, and 
> something that does the opposite of `:extend' for individual cases where 
> you want the new behavior?
>
> If not, why not?
>

I don't know, but I do not see this as a problem.  After all, if one wants 
the previous behavior, only a handful of faces need to be updated. 
Apparently the new behavior was considered better.  The NEWS item about 
this (in NEWS.27) states:  "This is to make Emacs behave more like other 
GUI applications with respect to displaying faces that cross line 
boundaries."

Gregory

bug#42347: Feature request: Visual block attribute for overlays

[Drew Adams]

> >> The previous behavior can be obtained with the ":extend t" face
> >> attribute.
> >
> > Only on an individual basis, right?  If you want to get the previous
> > behavior everywhere, do you need to change every overlay?
> 
> AFAIK, yes.
> 
> > Or is there an option for that?  If not, why not?
> >
> > And is there a way to get the previous behavior by default, and
> > something that does the opposite of `:extend' for individual cases
> > where you want the new behavior?  If not, why not?
> >
> 
> I don't know, but I do not see this as a problem.  After all, if one
> wants the previous behavior, only a handful of faces need to be updated.

Really?  That's what people thought at the beginning
of this change.  Then more popped up, one by one.

Why should a user, and not necessarily a lisper, need
to find and fix each such face - handful or not - to
get back the previous behavior, if that's what s?he
prefers?

> Apparently the new behavior was considered better.  The NEWS item about
> this (in NEWS.27) states:  "This is to make Emacs behave more like
> other GUI applications with respect to displaying faces that cross line
> boundaries."

Yes, I know the rationale.  It doesn't follow that
all, or even most, users feel the same way.

There are lots of outside-Emacs behaviors that we
don't impose as the default - let alone the only -
behavior in Emacs.

I'm not making an argument that users shouldn't be
able to get the new behavior, or even that the new
behavior should not have been adopted immediately
as the default (well...).  My argument is to make
it simple for users to switch behaviors.  Why not?

bug#42307: Feature request: Visual block attribute for overlays

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>
>> I don't know, but I do not see this as a problem.  After all, if one 
>> wants the previous behavior, only a handful of faces need to be 
>> updated.
>
> Really?  That's what people thought at the beginning of this change. 
> Then more popped up, one by one.
>
> Why should a user, and not necessarily a lisper, need to find and fix 
> each such face - handful or not - to get back the previous behavior, if 
> that's what s?he prefers?
>

I do not like the new default behavior either, but I can understand the 
viewpoint of those who decided that it would be the new default, even if I 
do not necessarily agree with their reasons.  I would have been shocked if 
the old behavior was not available anymore, but that isn't the case, so I 
don't think it's a problem.

>
>> Apparently the new behavior was considered better.  The NEWS item about 
>> this (in NEWS.27) states:  "This is to make Emacs behave more like 
>> other GUI applications with respect to displaying faces that cross line 
>> boundaries."
>
> Yes, I know the rationale.  It doesn't follow that all, or even most, 
> users feel the same way.
>
> There are lots of outside-Emacs behaviors that we don't impose as the 
> default - let alone the only - behavior in Emacs.
>
> I'm not making an argument that users shouldn't be able to get the new 
> behavior, or even that the new behavior should not have been adopted 
> immediately as the default (well...).  My argument is to make it simple 
> for users to switch behaviors.  Why not?
>

Yes, I agree with you that a way to make the old behavior the default one 
again would be welcome.  I guess this is doable, but from my point of view 
it is not a priority.  I would rather prefer to see the feature I proposed 
implemented, I think it would be a real improvement.

Gregory