bug#42307: Feature request: Visual block attribute for overlays

bug#42307: Feature request: Visual block attribute for overlays

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[-- Attachment #1: Type: text/plain, Size: 1453 bytes --]


In Emacs 21 to 26, overlays between two points in the buffer on two 
different lines extend to the right border of the window.  This has 
changed in Emacs 27, and now the default is that overlays extend only one 
character position after the last character of the line.  The previous 
behavior can be obtained with the ":extend t" face attribute. I agree that 
the earlier behavior was not optimal, but I think the current behavior 
with its staircase aspect is (at least for some cases) not optimal either.

I think a third way to display overlays would make sense, and would be 
better than the earlier and current defaults.  Let's name this attribute 
":visualblock".  It would produce the following result:

1. calculate the overlay as it would have been displayed by Emacs 21-26.

2. remove all pixel columns on the right *and on the left* of the overlay 
which have no "content" (that is, no characters on the right, and 
whitespace characters on the left).

With this, on overlay on, for example, a block of code between () or {} 
would be displayed on the screen as a block.  The above "algorithm" works 
with fixed and variable-width fonts, but could be made more efficient for 
fixed-width fonts.

To give an example of what I mean, I attach three pictures, taken with 
(show-paren-mode) and (setq show-paren-style 'expression), showing 
respectively the earlier default, the current default, and the proposed 
"visualblock" feature.

Gregory

[-- Attachment #2: Type: image/png, Size: 210270 bytes --]

[-- Attachment #3: Type: image/png, Size: 210486 bytes --]

[-- Attachment #4: Type: image/png, Size: 210558 bytes --]

bug#42307: Feature request: Visual block attribute for overlays

[Drew Adams]

> I think a third way to display overlays would make sense, and would be
> better than the earlier and current defaults.  Let's name this
> attribute ":visualblock".  ...
>
> With this, on overlay on, for example, a block of code between () or {}
> would be displayed on the screen as a block.  The above "algorithm"
> works with fixed and variable-width fonts, but could be made more
> efficient for fixed-width fonts.
> 
> To give an example of what I mean, I attach three pictures, taken with
> (show-paren-mode) and (setq show-paren-style 'expression), showing
> respectively the earlier default, the current default, and the proposed
> "visualblock" feature.

Interesting.  Thanks for the images, in particular.
I agree that each of the 3 is useful, and they all
should be available.

Looking only at the images, I have this question about
your "visual block" coverage:

What if `overlay-start' were on the `u', instead of the
`i', of `if (consp ,funs))'?  Would the overlay cover
only from that `u' onward, or would it still cover from
the `i' of `if' onward?  IOW, does the left edge of the
highlighted area extend downward from `overlay-start',
or does it start from the first non-whitespace char in
the line?

What about a variant of your "visual block" that
extends the overlay so that all lines, from the first
line, which contains `overlay-start' to the last line,
which contains `overlay-end', are covered through the
same columns?  Coverage (highlighting) would then
always be a rectangle.  In your example, it would
the highlighting of the last line would be extended
to the same column as that of the other lines.

Your "visual block", and the variant just described,
just like the old behavior, cover both whitespace
chars and empty screen real estate (no chars), from
`overlay-start' to `overlay-end'.  The new behavior
is the only one that covers only _chars_ in that
span (whitespace or other chars).

In any case, whatever the choices we offer, Elisp
should make it simple to choose any of them.

bug#42307: Feature request: Visual block attribute for overlays

[Drew Adams]

> Interesting.  Thanks for the images, in particular.
> I agree that each of the 3 is useful, and they all
> should be available.

I should have said they all should be considered.

bug#42307: Feature request: Visual block attribute for overlays

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Thanks for your feedback!

>
> Looking only at the images, I have this question about your "visual 
> block" coverage:
>
> What if `overlay-start' were on the `u', instead of the `i',
>

Actually it starts on the '(', the opening parenthesis is green if you 
look close enough.

>
> of `if (consp ,funs))'?  Would the overlay cover only from that `u' 
> onward, or would it still cover from the `i' of `if' onward?  IOW, does 
> the left edge of the highlighted area extend downward from 
> `overlay-start', or does it start from the first non-whitespace char in 
> the line?
>

If you take the algorithm, it would start on the "u", extend to the place 
it extends on the picture on the right, and on the next lines two 
whitespace characters on the left would not be displayed as green anymore 
(that is, the green area would start under the "f" of "(if".

>
> What about a variant of your "visual block" that extends the overlay so 
> that all lines, from the first line, which contains `overlay-start' to 
> the last line, which contains `overlay-end', are covered through the 
> same columns?  Coverage (highlighting) would then always be a rectangle. 
> In your example, it would the highlighting of the last line would be 
> extended to the same column as that of the other lines.
>

That's a possible variant, indeed.  Let's name this one "visualrectangle". 
But from a programmer's point of view it makes (IMHO) less sense.

>
> Your "visual block", and the variant just described, just like the old 
> behavior, cover both whitespace chars and empty screen real estate (no 
> chars), from `overlay-start' to `overlay-end'.  The new behavior is the 
> only one that covers only _chars_ in that span (whitespace or other 
> chars).
>

Indeed.  Yet another possibility would be cover only non-whitespace chars. 
Let's name this one "visiblechars".

>
> In any case, whatever the choices we offer, Elisp should make it simple 
> to choose any of them.
>

Yes ;-)  My personal preference order among the possible choices is:

1. visualblock
2. the previous default
3. visiblechars
4. the current default
5. visualrectangle

Gregory

bug#42307: Feature request: Visual block attribute for overlays

[Drew Adams]

> > What if `overlay-start' were on the `u', instead of the `i',
> 
> Actually it starts on the '(', the opening parenthesis is green if you
> look close enough.

Yes, that's what I meant and should have said.

> > of `if (consp ,funs))'?  Would the overlay cover only from that `u'
> > onward, or would it still cover from the `i' of `if' onward?  IOW,
> > does the left edge of the highlighted area extend downward from
> > `overlay-start', or does it start from the first non-whitespace
> > char in the line?
> 
> If you take the algorithm, it would start on the "u", extend to the
> place it extends on the picture on the right, and on the next lines
> two whitespace characters on the left would not be displayed as
> green anymore (that is, the green area would start under the "f" of
> "(if".

Why is that? That starting position of the second line's
highlight doesn't correspond to anything "natural", does
it?  I would expect either that it starts as it does now,
directly under the `(' of the first line, or directly
under the `u' that started the overlay.

> > What about a variant of your "visual block" that extends the
> > overlay so that all lines, from the first line, which contains
> > `overlay-start' to the last line, which contains `overlay-end',
> > are covered through the same columns?  Coverage (highlighting)
> > would then always be a rectangle.  In your example, the
> > highlighting of the last line would be extended to the same
> > column as that of the other lines.
> >
> 
> That's a possible variant, indeed.  Let's name this one
> "visualrectangle".  But from a programmer's point of view it
> makes (IMHO) less sense.

Neither makes sense from a coding point of view, I think.
At least not what you describe above, where the second
line's highlighting starts under the `f' of the first
line's `if'.  What does that position even correspond to?

In any case, overlays, and highlighting, are not just for
code.

> > Your "visual block", and the variant just described, just
> > like the old behavior, cover both whitespace chars and
> > empty screen real estate (no chars), from `overlay-start'
> > to `overlay-end'.  The new behavior is the only one that
> > covers only _chars_ in that span (whitespace or other chars).
> >
> 
> Indeed.  Yet another possibility would be cover only
> non-whitespace chars. Let's name this one "visiblechars".
> 
> > In any case, whatever the choices we offer, Elisp should
> > make it simple to choose any of them.
> >
> 
> Yes ;-)  My personal preference order among the possible choices is:
> 
> 1. visualblock
> 2. the previous default
> 3. visiblechars
> 4. the current default
> 5. visualrectangle

bug#42307: Feature request: Visual block attribute for overlays

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[-- Attachment #1: Type: text/plain, Size: 1377 bytes --]


>
>> If you take the algorithm, it would start on the "u", extend to the 
>> place it extends on the picture on the right, and on the next lines two 
>> whitespace characters on the left would not be displayed as green 
>> anymore (that is, the green area would start under the "f" of "(if".
>
> Why is that? That starting position of the second line's highlight 
> doesn't correspond to anything "natural", does it?  I would expect 
> either that it starts as it does now, directly under the `(' of the 
> first line, or directly under the `u' that started the overlay.
>

In fact it does, the overlay goes upwards above the comment below the "f" 
of "(if".  I attach a new picture between two arbitrary points that are 
not brackets.  I don't see any other way to define in simple terms how the 
whitespace on the left should be trimmed.  The "visualblock" would thus 
highlight all the text of the block between overlay-begin and overlay-end, 
and would be a rectangle except for the first and last lines, that is, 
except before overlay-begin and after overlay-end.

>
> In any case, overlays, and highlighting, are not just for code.
>

Indeed.  I think the proposed "visualblock" attribute is general and 
simple enough that it could be used not only for code.  E.g. it can be 
used to highlight a portion of text with whitespace on the left, a TODO 
item, say.

Gregory

[-- Attachment #2: Type: image/png, Size: 212444 bytes --]

bug#42307: Feature request: Visual block attribute for overlays

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[-- Attachment #1: Type: text/plain, Size: 185 bytes --]


>
> Yet another possibility would be cover only non-whitespace chars. Let's 
> name this one "visiblechars".
>

FTR, here is a visual representation of that other possibility.

Gregory

[-- Attachment #2: Type: image/png, Size: 210619 bytes --]

bug#42307: Feature request: Visual block attribute for overlays

[Juri Linkov]

>> What about a variant of your "visual block" that extends the overlay so
>> that all lines, from the first line, which contains `overlay-start' to
>> the last line, which contains `overlay-end', are covered through the same
>> columns?  Coverage (highlighting) would then always be a rectangle. In
>> your example, it would the highlighting of the last line would be
>> extended to the same column as that of the other lines.
>
> That's a possible variant, indeed.  Let's name this one
> "visualrectangle". But from a programmer's point of view it makes (IMHO)
> less sense.

How this highlighting is different from the rectangle selection mode
activated by 'C-x SPC'?

bug#42307: Feature request: Visual block attribute for overlays

[Drew Adams]

> How this highlighting is different from the rectangle selection mode
> activated by 'C-x SPC'?

I'm not the OP, but I think the point is to provide
different behaviors for overlays, just as we now
have the original behavior (rebaptized as :extend)
and the new behavior that doesn't extend.

It's not about text selection or defining the region.
It's about overlay properties.  That's my understanding,
anyway.

bug#42307: Feature request: Visual block attribute for overlays

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>
>> How this highlighting is different from the rectangle selection mode 
>> activated by 'C-x SPC'?
>
> I'm not the OP, but I think the point is to provide different behaviors 
> for overlays, just as we now have the original behavior (rebaptized as 
> :extend) and the new behavior that doesn't extend.
>
> It's not about text selection or defining the region. It's about overlay 
> properties.  That's my understanding, anyway.
>

Your understanding is correct, the point is not to create a new selection 
mode (they all exist as far as I can see) but a new overlay behavior, with 
a face attribute.

Gregory

bug#42307: Feature request: Visual block attribute for overlays

[Juri Linkov]

>> How this highlighting is different from the rectangle selection mode
>> activated by 'C-x SPC'?
>
> I'm not the OP, but I think the point is to provide
> different behaviors for overlays, just as we now
> have the original behavior (rebaptized as :extend)
> and the new behavior that doesn't extend.
>
> It's not about text selection or defining the region.
> It's about overlay properties.  That's my understanding,
> anyway.

This is exactly what I meant.  I only asked if it should look like
the highlighting created by rectangle selection mode where the
beginning of the overlay is under point and its left border goes down
on the same column to the last line of the overlay.

More generally the question is: what rules define the column
of the left vertical border and its right border?
I see at least these possible options:

1. for left vertical border:
- the same column where point currently is located;
- the leftmost column with non-whitespace character

2. for right vertical border:
- the rightmost column with non-whitespace character;
- something more?

bug#42307: Feature request: Visual block attribute for overlays

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>
> This is exactly what I meant.  I only asked if it should look like the 
> highlighting created by rectangle selection mode where the beginning of 
> the overlay is under point and its left border goes down on the same 
> column to the last line of the overlay.
>
> More generally the question is: what rules define the column of the left 
> vertical border and its right border?
>

Did you actually read anything of the above, or did you only read the 
title?  I explained what the rules are, and gave two visual examples to 
illustrate the rules.

Gregory

bug#42307: Feature request: Visual block attribute for overlays

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>
>> I don't know, but I do not see this as a problem.  After all, if one 
>> wants the previous behavior, only a handful of faces need to be 
>> updated.
>
> Really?  That's what people thought at the beginning of this change. 
> Then more popped up, one by one.
>
> Why should a user, and not necessarily a lisper, need to find and fix 
> each such face - handful or not - to get back the previous behavior, if 
> that's what s?he prefers?
>

I do not like the new default behavior either, but I can understand the 
viewpoint of those who decided that it would be the new default, even if I 
do not necessarily agree with their reasons.  I would have been shocked if 
the old behavior was not available anymore, but that isn't the case, so I 
don't think it's a problem.

>
>> Apparently the new behavior was considered better.  The NEWS item about 
>> this (in NEWS.27) states:  "This is to make Emacs behave more like 
>> other GUI applications with respect to displaying faces that cross line 
>> boundaries."
>
> Yes, I know the rationale.  It doesn't follow that all, or even most, 
> users feel the same way.
>
> There are lots of outside-Emacs behaviors that we don't impose as the 
> default - let alone the only - behavior in Emacs.
>
> I'm not making an argument that users shouldn't be able to get the new 
> behavior, or even that the new behavior should not have been adopted 
> immediately as the default (well...).  My argument is to make it simple 
> for users to switch behaviors.  Why not?
>

Yes, I agree with you that a way to make the old behavior the default one 
again would be welcome.  I guess this is doable, but from my point of view 
it is not a priority.  I would rather prefer to see the feature I proposed 
implemented, I think it would be a real improvement.

Gregory

bug#42307: Feature request: Visual block attribute for overlays

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>>> and \n is the newline, so it isn't on the same line.
>>>
>>
>> It is, but it's the last character of the line.  With the current 
>> default behavior it is displayed, there is one more blank character 
>> after each line, just type C-x = on that character.
>
> My point is that \n cannot be "leading space" of a line.  It is the end 
> of the previous line.
>

Yes, now I see what you mean.  So the regexp for "blank" on the left would 
be "[ \t]*", and the regexp for "blank" on the right would be "\n%*", 
where "%" denotes "no character".

>
>>> And finally, what about stretches of whitespace generated by the 
>>> 'space' display properties?
>>
>> I don't know, and I'm not sure I fully understand the question.
>
> The point is that the result of displaying these properties is exactly 
> the same as tabs and spaces.  So excluding them would surprise users.
>

Okay.  So... let's include them ;-)

In fact, I'm not entirerly sure that explaining (or implementing) that 
behavior with regexpes or character is the best thing to do.  It's a 
matter of visual representation.  Perhaps the following explanation is 
clearer (or more precise): draw the overlay with :extend t, and remove 
pixel columns on the left and on the right that are displayed in the same 
way as a whitespace character would have been displayed.

Gregory

bug#42307: Feature request: Visual block attribute for overlays

[Eli Zaretskii]

> Date: Tue, 14 Jul 2020 17:01:43 +0000
> From: Gregory Heytings via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> In fact, I'm not entirerly sure that explaining (or implementing) that 
> behavior with regexpes or character is the best thing to do.  It's a 
> matter of visual representation.  Perhaps the following explanation is 
> clearer (or more precise): draw the overlay with :extend t, and remove 
> pixel columns on the left and on the right that are displayed in the same 
> way as a whitespace character would have been displayed.

Well, it's not that simple.  For starters, there's no "pixel columns"
in the display engine, at least not on GUI frames.

More to the point, I think the feature you requested will be not very
simple to implement, because it requires the display code to do
look-ahead: when it lays out leading whitespace, it should consider
characters after that.  That calls for some complications, because the
display code basically inspects characters one by one and makes
(almost) all the decisions on the fly.

So someone will have fun implementing this.  Patches welcome.

bug#42307: Feature request: Visual block attribute for overlays

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>
>> In fact, I'm not entirerly sure that explaining (or implementing) that 
>> behavior with regexpes or character is the best thing to do.  It's a 
>> matter of visual representation.  Perhaps the following explanation is 
>> clearer (or more precise): draw the overlay with :extend t, and remove 
>> pixel columns on the left and on the right that are displayed in the 
>> same way as a whitespace character would have been displayed.
>
> Well, it's not that simple.  For starters, there's no "pixel columns" in 
> the display engine, at least not on GUI frames.
>

That's what I thought indeed.  My explanation was a specification, not an 
implementation hint ;-)

>
> More to the point, I think the feature you requested will be not very 
> simple to implement, because it requires the display code to do 
> look-ahead: when it lays out leading whitespace, it should consider 
> characters after that.  That calls for some complications, because the 
> display code basically inspects characters one by one and makes (almost) 
> all the decisions on the fly.
>
> So someone will have fun implementing this.  Patches welcome.
>

I'm not sure I have the necessary background to do this (it's a 
euphemism), but I'll try to see what I can do.

Gregory