bug#66216: 28.2; scroll-up-line doesn't work if there is a before-string overlay with newline

bug#66216: 28.2; scroll-up-line doesn't work if there is a before-string overlay with newline

[Herman, Géza]

This bug exists in 28.2, but on a not too old master as well.

Repro:
  - emacs -Q
  - M-: (overlay-put (make-overlay 72 72) 'before-string "Fake line\n")
  - this will put a "Fake line" at the 2nd line of the scratch buffer, 
(between the two default scratch buffer message lines)
  - M-x scroll-up-line
  - this will correctly scroll one line up
  - M-x scroll-up-line
  - this is the bug, no scroll happens

Also, if the overlay is added at the middle of some line (not at the 
beginning like in my repro steps), then scroll-up-line will "scroll" 
until the overlay only, making the overlay to visually move the left 
side. Then further scroll-up-line commands won't have an effect.

bug#66216: 28.2; scroll-up-line doesn't work if there is a before-string overlay with newline

[Eli Zaretskii]

> Date: Tue, 26 Sep 2023 20:30:53 +0200
> From: Herman@debbugs.gnu.org, Géza <geza.herman@gmail.com>
> 
> This bug exists in 28.2, but on a not too old master as well.
> 
> Repro:
>   - emacs -Q
>   - M-: (overlay-put (make-overlay 72 72) 'before-string "Fake line\n")
>   - this will put a "Fake line" at the 2nd line of the scratch buffer, 
> (between the two default scratch buffer message lines)
>   - M-x scroll-up-line
>   - this will correctly scroll one line up
>   - M-x scroll-up-line
>   - this is the bug, no scroll happens
> 
> Also, if the overlay is added at the middle of some line (not at the 
> beginning like in my repro steps), then scroll-up-line will "scroll" 
> until the overlay only, making the overlay to visually move the left 
> side. Then further scroll-up-line commands won't have an effect.

What do you expect to happen in these cases?

The scroll commands work by telling Emacs which buffer position to use
as the window-start for the next redisplay; then redisplay kicks in
and redraws the window using that starting position.  But for boring
technical reasons, Emacs is unable to start displaying a buffer from a
position where we have a before-string overlay, without displaying
that before-string first.  Which is what you see.

We could scroll one more line in these cases, but then the other group
of users will come up complaining that we scroll over too much text
(no one said the before-string must have only one newline, it could
have several ones, in which case we will scroll across all of those
lines).  So we punt and let the user invoke the scrolling commands
with an appropriate prefix argument; for example, in this case, just
say "C-u 2 M-x scroll-up-line RET", and Bob's your uncle.

But if this is not good enough, please tell what you'd like to see
instead, given the above limitation of the current display engine, and
maybe we will find better solutions for this conundrum.

Thanks.

bug#66216: 28.2; scroll-up-line doesn't work if there is a before-string overlay with newline

[Herman, Géza]

I have scroll-up/down-line mapped to a key. I'd like to see that if I 
press and hold these keys, Emacs can scroll continuously. It doesn't 
matter too much if at some position, it jumps 2 (or more) visible lines. 
Using prefix arguments is not a good solution for this case, because if 
I have a buffer with a lot of such overlays (like magit's blame buffer), 
it's very inconvenient that emacs stops very frequently, and then I have 
to nag it with a prefix argument to "please scroll further". But of 
course I can try to write some elisp function to work around this 
limitation.

As scroll-down-line jumps over overlays, so it already scrolls 2 visible 
lines in the mentioned case, it would make sense that scroll-up-line 
behaves the same as well. The current behavior is not consistent. I'd 
expect that if scroll-down-line moves window start somewhere, then 
scroll-up-line will undo this. But in this case scroll-down-line will 
move 2 lines, then scroll-up-line doesn't do anything.

On 9/29/23 13:28, Eli Zaretskii wrote:
>> Date: Tue, 26 Sep 2023 20:30:53 +0200
>> From: Herman@debbugs.gnu.org, Géza <geza.herman@gmail.com>
>>
>> This bug exists in 28.2, but on a not too old master as well.
>>
>> Repro:
>>    - emacs -Q
>>    - M-: (overlay-put (make-overlay 72 72) 'before-string "Fake line\n")
>>    - this will put a "Fake line" at the 2nd line of the scratch buffer,
>> (between the two default scratch buffer message lines)
>>    - M-x scroll-up-line
>>    - this will correctly scroll one line up
>>    - M-x scroll-up-line
>>    - this is the bug, no scroll happens
>>
>> Also, if the overlay is added at the middle of some line (not at the
>> beginning like in my repro steps), then scroll-up-line will "scroll"
>> until the overlay only, making the overlay to visually move the left
>> side. Then further scroll-up-line commands won't have an effect.
> What do you expect to happen in these cases?
>
> The scroll commands work by telling Emacs which buffer position to use
> as the window-start for the next redisplay; then redisplay kicks in
> and redraws the window using that starting position.  But for boring
> technical reasons, Emacs is unable to start displaying a buffer from a
> position where we have a before-string overlay, without displaying
> that before-string first.  Which is what you see.
>
> We could scroll one more line in these cases, but then the other group
> of users will come up complaining that we scroll over too much text
> (no one said the before-string must have only one newline, it could
> have several ones, in which case we will scroll across all of those
> lines).  So we punt and let the user invoke the scrolling commands
> with an appropriate prefix argument; for example, in this case, just
> say "C-u 2 M-x scroll-up-line RET", and Bob's your uncle.
>
> But if this is not good enough, please tell what you'd like to see
> instead, given the above limitation of the current display engine, and
> maybe we will find better solutions for this conundrum.
>
> Thanks.

bug#66216: 28.2; scroll-up-line doesn't work if there is a before-string overlay with newline

[Eli Zaretskii]

> Date: Fri, 29 Sep 2023 13:53:04 +0200
> Cc: 66216@debbugs.gnu.org
> From: Herman, Géza <geza.herman@gmail.com>
> 
> I have scroll-up/down-line mapped to a key. I'd like to see that if I 
> press and hold these keys, Emacs can scroll continuously. It doesn't 
> matter too much if at some position, it jumps 2 (or more) visible lines. 
> Using prefix arguments is not a good solution for this case, because if 
> I have a buffer with a lot of such overlays (like magit's blame buffer), 
> it's very inconvenient that emacs stops very frequently, and then I have 
> to nag it with a prefix argument to "please scroll further". But of 
> course I can try to write some elisp function to work around this 
> limitation.

I explained why catering only to this preference will bring us other
problems.

One solution is for you to write a simple function that scrolls by
more than 1 line if the basic scroll didn't change window-start
position.

And if this is a problem with Magit, then I would suggest that Magit
has its own scrolling function to overcome the problems.  I tried to
explain many times that using overlays and display properties in Magit
buffers was a bad idea, and the Magit developer even agreed with me at
some point.

> As scroll-down-line jumps over overlays, so it already scrolls 2 visible 
> lines in the mentioned case, it would make sense that scroll-up-line 
> behaves the same as well. The current behavior is not consistent. I'd 
> expect that if scroll-down-line moves window start somewhere, then 
> scroll-up-line will undo this. But in this case scroll-down-line will 
> move 2 lines, then scroll-up-line doesn't do anything.

The situation with scroll-down and scroll-up is not symmetric.

bug#66216: 28.2; scroll-up-line doesn't work if there is a before-string overlay with newline

[Herman, Géza]

On 9/29/23 17:28, Eli Zaretskii wrote:
> And if this is a problem with Magit, then I would suggest that Magit
> has its own scrolling function to overcome the problems.  I tried to
> explain many times that using overlays and display properties in Magit
> buffers was a bad idea, and the Magit developer even agreed with me at
> some point.
What other ways exist to achieve what magit blame does?

bug#66216: 28.2; scroll-up-line doesn't work if there is a before-string overlay with newline

[Eli Zaretskii]

> Date: Sat, 30 Sep 2023 19:09:29 +0200
> Cc: 66216@debbugs.gnu.org
> From: Herman, Géza <geza.herman@gmail.com>
> 
> 
> 
> On 9/29/23 17:28, Eli Zaretskii wrote:
> > And if this is a problem with Magit, then I would suggest that Magit
> > has its own scrolling function to overcome the problems.  I tried to
> > explain many times that using overlays and display properties in Magit
> > buffers was a bad idea, and the Magit developer even agreed with me at
> > some point.
> What other ways exist to achieve what magit blame does?

Populating a scratch buffer with generated text, for example.

bug#66216: 28.2; scroll-up-line doesn't work if there is a before-string overlay with newline

[Herman, Géza]

On 9/30/23 19:17, Eli Zaretskii wrote:
>
> Populating a scratch buffer with generated text, for example.
Then line numbers will be incorrect, and also the buffer won't be 
editable. And maybe there are other drawbacks of this approach. OK, I 
admit that editable blame buffers is not a very important feature, but I 
think that it's a good thing that line numbers are correct. And looking 
at this from the application developer's viewpoint: just because Emacs 
has limitation around overlays, they shouldn't choose a worse solution. 
Especially if one doesn't know about the limitation beforehand, but it 
emerges after 90% of the application is finished.

bug#66216: 28.2; scroll-up-line doesn't work if there is a before-string overlay with newline

[Eli Zaretskii]

> Date: Sat, 30 Sep 2023 19:29:51 +0200
> Cc: 66216@debbugs.gnu.org
> From: Herman, Géza <geza.herman@gmail.com>
> 
> On 9/30/23 19:17, Eli Zaretskii wrote:
> >
> > Populating a scratch buffer with generated text, for example.
> Then line numbers will be incorrect

Why not? it depends on how the buffer text is generated, doesn't it?

> and also the buffer won't be editable.

That is easily made possible from Lisp, isn't it?

> And maybe there are other drawbacks of this approach. OK, I 
> admit that editable blame buffers is not a very important feature, but I 
> think that it's a good thing that line numbers are correct.

I see no reason why line numbers should be incorrect.  The line
numbers in blame displays are produced by the VCS, not by Emacs, so
they should be correct no matter what.

> And looking at this from the application developer's viewpoint: just
> because Emacs has limitation around overlays, they shouldn't choose
> a worse solution.

A Lisp program written for Emacs definitely _should_ consider
limitations and restrictions of the Emacs infrastructure it uses, if
it wants to present a convenient, efficient, and well-looking
interface to the users.

> Especially if one doesn't know about the limitation beforehand, but
> it emerges after 90% of the application is finished.

I don't blame anyone, I'm just saying that this was explained years
ago.  In a nutshell, overlays in Emacs were never supposed to support
massive additions of text with newlines via overlay strings, they were
supposed to support relatively short strings that don't change the
line geometry too much.  Many of the original limitations were lifted
during the years, but that can be only done up to a point.

bug#66216: 28.2; scroll-up-line doesn't work if there is a before-string overlay with newline

[Herman, Géza]

On 9/30/23 19:37, Eli Zaretskii wrote:
>>
>> Then line numbers will be incorrect
> Why not? it depends on how the buffer text is generated, doesn't it?
This is an example of magit's default blame presentation mode: 
http://www.mycpu.org/images/magit-blame.jpeg

The header in front of each code chunk is an overlay. If we wanted to 
implement this without overlays (just simple text lines with text 
properties), then the headers will receive a line number, making the 
original line number incorrect.

I suppose this is what you meant by "populating a scratch buffer with 
generated text"

>> and also the buffer won't be editable.
> That is easily made possible from Lisp, isn't it?
How? The optimal solution is to edit the original buffer right away, 
just like how magit currently works. Maybe it's possible to sync between 
the scratch and the original buffer somehow. But this solution is 
awkward, and I'm sure it has a lot of pitfalls. Using overlays is a much 
more straightforward (from the package's viewpoint, and also from the 
user's).

> A Lisp program written for Emacs definitely _should_ consider
> limitations and restrictions of the Emacs infrastructure it uses, if
> it wants to present a convenient, efficient, and well-looking
> interface to the users.
Fair enough. Are these limitations documented?

Multiple-line overlays seem to work well in a lot of cases, several 
packages use it. It is just there are some cases where they don't work 
how one may expect. So one might think that such overlays are fully 
supported, and these cases are just bugs that can be fixed without too 
much work.

bug#66216: 28.2; scroll-up-line doesn't work if there is a before-string overlay with newline

[Eli Zaretskii]

> Date: Sat, 30 Sep 2023 21:15:49 +0200
> Cc: 66216@debbugs.gnu.org
> From: Herman, Géza <geza.herman@gmail.com>
> 
> 
> 
> On 9/30/23 19:37, Eli Zaretskii wrote:
> >>
> >> Then line numbers will be incorrect
> > Why not? it depends on how the buffer text is generated, doesn't it?
> This is an example of magit's default blame presentation mode: 
> http://www.mycpu.org/images/magit-blame.jpeg
> 
> The header in front of each code chunk is an overlay. If we wanted to 
> implement this without overlays (just simple text lines with text 
> properties), then the headers will receive a line number, making the 
> original line number incorrect.

I see no line numbers in the screenshot you posted.

If you mean the line number displayed in the mode line, then this is
just one way of showing line numbers, and it just happens to match the
line numbers of the source file when overlay strings are used.  Other
methods of line number might not behave like that: for example, if you
set display-line-numbers to the value 'visual', the "fake" newlines
produced by overlays will be counted as legitimate lines, and the line
counts will be wrong.

My conclusion is that relying on line-number correspondence is
fragile, and for best results the blame display should show the line
numbers produced by Git.

> >> and also the buffer won't be editable.
> > That is easily made possible from Lisp, isn't it?
> How? The optimal solution is to edit the original buffer right away, 
> just like how magit currently works. Maybe it's possible to sync between 
> the scratch and the original buffer somehow. But this solution is 
> awkward, and I'm sure it has a lot of pitfalls.

Well, the built-in VC mode solves all those issues nicely without
using any overlays.  So it isn't as difficult as you seem to think.

> > A Lisp program written for Emacs definitely _should_ consider
> > limitations and restrictions of the Emacs infrastructure it uses, if
> > it wants to present a convenient, efficient, and well-looking
> > interface to the users.
> Fair enough. Are these limitations documented?

Those that are important are indeed documented.

> Multiple-line overlays seem to work well in a lot of cases, several 
> packages use it. It is just there are some cases where they don't work 
> how one may expect. So one might think that such overlays are fully 
> supported, and these cases are just bugs that can be fixed without too 
> much work.

Like I said, a simple solution is for Magit to override the default
definition of scroll-up-line.

bug#66216: 28.2; scroll-up-line doesn't work if there is a before-string overlay with newline

[Herman, Géza]

On 10/1/23 10:51, Eli Zaretskii wrote:
> If you mean the line number displayed in the mode line, then this is
> just one way of showing line numbers, and it just happens to match the
> line numbers of the source file when overlay strings are used.  Other
> methods of line number might not behave like that: for example, if you
> set display-line-numbers to the value 'visual', the "fake" newlines
> produced by overlays will be counted as legitimate lines, and the line
> counts will be wrong.
Yes, I meant that. If I want to know the line number of a particular 
line, then I use the indicator on the modeline. I wouldn't use 
display-line-numbers with 'visual, because it may not tell this number. 
With your suggestion, it's not possible to tell the line number. With 
magit working with overlays, it is. It's not "just happens to match", 
but designed this way.

To me, it's clear that using overlays for this problem is a better 
solution than using a separate buffer. So far, you didn't say a single 
thing for which using a separate buffer is better. You're just 
suggesting this as workaround for Emacs's limitation. But it's an 
inferior solution. In an ideal world, not every package (which use 
newline-containing overlays) should accomodate to Emacs's limitation, 
but Emacs would be fixed to handle newline-containing overlays better. 
Note: I know that Emacs is free, developed by volunteers, etc., I don't 
want to be sound like someone who tells how to develop Emacs. But in the 
long term, this is the forward-looking solution. I know, someone has to 
do it, it's a large investment.

> My conclusion is that relying on line-number correspondence is
> fragile, and for best results the blame display should show the line
> numbers produced by Git.
I haven't noticed any problems how magit-blame works in this regard. 
Magit just decorates the buffer. And Emacs should be able to tell for a 
particular line its line number. This should be the easiest part, as 
Emacs doesn't have to do anything special, just tell the line number of 
the point (ignoring all overlays). Why is it fragile?


>> How? The optimal solution is to edit the original buffer right away,
>> just like how magit currently works. Maybe it's possible to sync between
>> the scratch and the original buffer somehow. But this solution is
>> awkward, and I'm sure it has a lot of pitfalls.
> Well, the built-in VC mode solves all those issues nicely without
> using any overlays.  So it isn't as difficult as you seem to think.
Do you mean vc-annotate, or something else? Is it possible to edit the 
annotate buffer? And blame information put in a header above the code 
chunk, just like magit's? How can I achieve these?


>> Fair enough. Are these limitations documented?
> Those that are important are indeed documented.
I'm not sure what's documented and what's not. But, I think the fact 
that Emacs has limitations around newline-containing overlays/display 
text-properties is important. So if this is not documented, then it 
should be. Just I alone discovered 3-4 problems with it. Emacs has 
limitations, the manual can have a subsection to list them with a short 
description. If this limitation was documented, this bug and our whole 
discussion may not exist, saving time for both of us. Not just now, but 
I had previous similar bug reports. But now I learned the hard way that 
if Emacs has a bug related to newline-containing overlays, I should not 
report it.