bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Stefan Monnier]

Package: Emacs
Version: 28.0.50


    % src/emacs -Q --eval '(setq max-mini-window-height 1)' -f icomplete-mode
    M-x a

at this point, you should presumably not see the "M-x a" in your
minibuffer window but only something of the form "{rp | lign | ..."

This is probably related to bug#24293 and bug#39379.


        Stefan

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Sat, 19 Sep 2020 13:54:13 -0400
> 
>     % src/emacs -Q --eval '(setq max-mini-window-height 1)' -f icomplete-mode
>     M-x a
> 
> at this point, you should presumably not see the "M-x a" in your
> minibuffer window but only something of the form "{rp | lign | ..."

Seems like a bug in icomplete: it attempts to compute the maximum
length of candidates to be displayed, but seems like it fails, because
the single mini-window line is continued, with no ellipsis at the end
of the visible line?

The fact that it calls window-width with no arguments is one possible
problem -- it assumes the default face's font.  But I think the
problem is more prominent than just that.

It should produce an overlay string that fits in the window, then the
prompt will be visible.

Am I missing something?

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Stefan Monnier]

>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Date: Sat, 19 Sep 2020 13:54:13 -0400
>> 
>>     % src/emacs -Q --eval '(setq max-mini-window-height 1)' -f icomplete-mode
>>     M-x a
>> 
>> at this point, you should presumably not see the "M-x a" in your
>> minibuffer window but only something of the form "{rp | lign | ..."
>
> Seems like a bug in icomplete: it attempts to compute the maximum
> length of candidates to be displayed, but seems like it fails, because
> the single mini-window line is continued, with no ellipsis at the end
> of the visible line?

I disagree: icomplete merely added text after point via an overlay and
didn't do anything which explicitly justifies horizontal scrolling.

I suspect the problem is that point is right on the overlay, so in
a sense it's both before *and* after the "{...}" text.  Conceptually it
should be considered as being before (which is why the cursor is placed
on the `{`), and the redisplay somewhat agrees with it (because it
hides the end of "{...}" rather than its beginning) but not completely
since it scrolled the display even though the `{` was already visible
without it.

> The fact that it calls window-width with no arguments is one possible
> problem -- it assumes the default face's font.  But I think the
> problem is more prominent than just that.
>
> It should produce an overlay string that fits in the window, then the
> prompt will be visible.

That would merely work around the underlying problem (and as you know
it's wickedly difficult to construct a string which will have "just the
right size" to fit into the minibuffer window).

Maybe there's a good reason for the redisplay to behave this way, but if
that's the case we need some way for icomplete (and other similar cases)
to make it behave differently.


        Stefan

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: 43519@debbugs.gnu.org
> Date: Sat, 19 Sep 2020 15:42:12 -0400
> 
> > Seems like a bug in icomplete: it attempts to compute the maximum
> > length of candidates to be displayed, but seems like it fails, because
> > the single mini-window line is continued, with no ellipsis at the end
> > of the visible line?
> 
> I disagree: icomplete merely added text after point via an overlay and
> didn't do anything which explicitly justifies horizontal scrolling.

Maybe I'm missing something, but what does the code in
icomplete-completions that calls string-width and window-width try to
do, then?  I mean this part:

	     ;;"-prospects" - more than one candidate
	     (prospects-len (+ (string-width
				(or determ (concat open-bracket close-bracket)))
			       (string-width icomplete-separator)
			       (+ 2 (string-width ellipsis)) ;; take {…} into account
			       (string-width (buffer-string))))
             (prospects-max
              ;; Max total length to use, including the minibuffer content.
              (* (+ icomplete-prospects-height
                    ;; If the minibuffer content already uses up more than
                    ;; one line, increase the allowable space accordingly.
                    (/ prospects-len (window-width)))
                 (window-width)))

> I suspect the problem is that point is right on the overlay, so in
> a sense it's both before *and* after the "{...}" text.

The point is at EOB, and we have an overlay string there.  The overlay
string has a 'cursor' property on its first character, so the display
engine puts the cursor there.  Without that, we'd have the cursor at
the end of the overlay string, thus showing something even less
reasonable.

But I'm not sure how this is related to the issue at hand.

> > It should produce an overlay string that fits in the window, then the
> > prompt will be visible.
> 
> That would merely work around the underlying problem

What do you think is the underlying problem?

> (and as you know it's wickedly difficult to construct a string which
> will have "just the right size" to fit into the minibuffer window).

It doesn't have to be "just the right size", it could err on the safe
side.  It already attempts to do so, by avoiding truncation in the
middle of a candidate.  It should just do a better job, that's all.

> Maybe there's a good reason for the redisplay to behave this way

Behave in what way? what's special about what you see on display in
this case, given the contents of the mini-window's buffer?

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Stefan Monnier]

>> I disagree: icomplete merely added text after point via an overlay and
>> didn't do anything which explicitly justifies horizontal scrolling.
>
> Maybe I'm missing something, but what does the code in
> icomplete-completions that calls string-width and window-width try to
> do, then?  I mean this part:
>
> 	     ;;"-prospects" - more than one candidate
> 	     (prospects-len (+ (string-width
> 				(or determ (concat open-bracket close-bracket)))
> 			       (string-width icomplete-separator)
> 			       (+ 2 (string-width ellipsis)) ;; take {…} into account
> 			       (string-width (buffer-string))))
>              (prospects-max
>               ;; Max total length to use, including the minibuffer content.
>               (* (+ icomplete-prospects-height
>                     ;; If the minibuffer content already uses up more than
>                     ;; one line, increase the allowable space accordingly.
>                     (/ prospects-len (window-width)))
>                  (window-width)))

That's not relevant to the issue at hand.  I used `icomplete-mode` only
as a vehicle to show the underlying behavior with a short recipe which
exhibits a real-life problem.

>> > It should produce an overlay string that fits in the window, then the
>> > prompt will be visible.
>> That would merely work around the underlying problem
> What do you think is the underlying problem?

That the redisplay performed horizontal scrolling when it was not needed
since the cursor was already visible without such scrolling.

>> (and as you know it's wickedly difficult to construct a string which
>> will have "just the right size" to fit into the minibuffer window).
> It doesn't have to be "just the right size", it could err on the safe
> side.  It already attempts to do so, by avoiding truncation in the
> middle of a candidate.  It should just do a better job, that's all.

And how do we generalize that to the case where the overlay contains
newlines, TABs, chars in different scripts using different fonts,
different faces, images, etc.... ?

>> Maybe there's a good reason for the redisplay to behave this way
> Behave in what way? what's special about what you see on display in
> this case, given the contents of the mini-window's buffer?

Try the recipe below instead:

    (minibuffer-with-setup-hook
        (lambda ()
          (insert "hello")
          (let ((ol (make-overlay (point) (point)))
                (max-mini-window-height 1)
                (text "askdjfhaklsjdfhlkasjdfhklasdhflkasdhflkajsdhflkashdfkljahsdlfkjahsdlfkjhasldkfhalskdjfhalskdfhlaksdhfklasdhflkasdhflkasdhflkajsdhklajsdgh"))
            (save-excursion (insert text))
            (sit-for 2)
            (delete-region (point) (point-max))
            (put-text-property 0 1 'cursor t text)
            (overlay-put ol 'after-string text)
            (sit-for 2)
            (delete-overlay ol)))
      (read-string "toto: "))

This performs "display of text after point" in 2 different ways:
- first by `insert`.
- then with an overlay.
The visual rendering of the text is the same, with the cursor at the
same place.  When we do it with `insert` there is no horizontal
scrolling, but when we do it with an overlay the text gets scrolled so
the cursor is at `window-start`.

`icomplete` needs the behavior to be the same as with `insert`, but it
prefers to use an overlay to avoid some undesirable side-effects of
modifying the actual text.

So the question is: how to get the same behavior as what we'd get with
`insert` but without actually modifying the buffer's contents?

Is it more clear now?


        Stefan

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> For example, with (setq icomplete-separator "\n"), after M-x a the 
>> contents of the minibuffer is displayed as follows:
>>
>> {rp
>> lign
>> propos
>> sm-mode
>> <... other candidates, one on each line>
>
> Does the entire list of candidates fit in the (enlarged) mini-window in 
> that case? or are some of the candidates not displayed due to lack of 
> space?
>

The list of candidates is in this case (with emacs -Q) 32 lines long 
("{rp" on the first line, ... "...}" on the last line).  With (setq 
max-mini-window-height 32) the M-x prompt is displayed, with (setq 
max-mini-window-height 31) it isn't.

But the point is that there is clearly enough horizontal space, so there 
is (or there does not seem to be) no reason to scroll horizontally.

>> After pressing C-b (or <left>), the prompt becomes visible, and the 
>> minibuffer is displayed as follows:
>
> C-b moves point from EOB to a character before EOB, thus the difference 
> (AFAIR).
>

Yes, I know what C-b does, what I find surprising (but perhaps it is not, 
I'm not an expert) is that only one line (the one with the prompt) is 
scrolled horizontally.

[taken from emacs-devel:]
>
> You are asking the display engine to do the impossible.
>

No.  With a minibuffer-only frame, the bug is not present.  Whatever the 
value of icomplete-separator (that is, with either a horizontal or 
vertical list of candidates), you do see the "M-x" prompt and the 
characters typed so far, even with (setq max-mini-window-height 1) and a 
one line minibuffer.  So it's feasible, and the display engine knows how 
to do it.

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: 43519@debbugs.gnu.org
> Date: Sat, 19 Sep 2020 18:06:20 -0400
> 
> > 	     ;;"-prospects" - more than one candidate
> > 	     (prospects-len (+ (string-width
> > 				(or determ (concat open-bracket close-bracket)))
> > 			       (string-width icomplete-separator)
> > 			       (+ 2 (string-width ellipsis)) ;; take {…} into account
> > 			       (string-width (buffer-string))))
> >              (prospects-max
> >               ;; Max total length to use, including the minibuffer content.
> >               (* (+ icomplete-prospects-height
> >                     ;; If the minibuffer content already uses up more than
> >                     ;; one line, increase the allowable space accordingly.
> >                     (/ prospects-len (window-width)))
> >                  (window-width)))
> 
> That's not relevant to the issue at hand.  I used `icomplete-mode` only
> as a vehicle to show the underlying behavior with a short recipe which
> exhibits a real-life problem.

TL;DR: Please describe those real-life problems in more detail.  I
hope my explanations below clarify why this is needed.

Details:

If you want the display engine to behave with after-strings the same
as with buffer text in this and similar cases, then this is AFAIU
impossible under the current design of the display code.  In
particular, functions that allow layout decisions by simulating
display treat overlay strings as a single unbreakable chunk of text,
and will not stop inside such strings, and so cannot provide the same
information they do when the text in the mini-window comes from a
buffer.  So the code in resize_mini_window and elsewhere cannot
possibly behave the same in the two variants of mini-window display
you provided in your test case.  Or at least I don't know how to make
it behave the same.

IOW, to the best of my knowledge and understanding, this is not a bug,
but a direct consequence of how the display code was designed.

Therefore, we are left with 2 possibilities:

  . Fix these problems on the application level.  In the case in
    point, that would mean for icomplete.el to augment its
    calculations of where to truncate the list of candidates so that
    the problem doesn't happen (and AFAICT it doesn't happen if the
    stuff to be displayed in the mini-window fits the mini-window
    after resizing it to max-mini-window-height).

  . Define in more detail what situations we would like to fix in the
    display code, so that we could install special ad-hoc changes
    there to handle those situations.  For example: is it true that in
    all of these situations starting the mini-window display at BOB
    would DTRT?  If so, I think this could be arranged.  If not, why
    not, and what is the more correct definition of the situations we
    want to handle?

> > What do you think is the underlying problem?
> 
> That the redisplay performed horizontal scrolling when it was not needed
> since the cursor was already visible without such scrolling.

There's no horizontal scrolling.  The issue is with determining the
mini-window's start position.  In the case with the overlay, we
compute that position as EOB, whereas in the case with buffer text, we
compute it to be at BOB.  The reason is what I said: the very
different behavior of the move_it_* functions when they need to
traverse overlay strings.

The basic logic of resize_mini_window is like this:

  . compute the number of screen lines required for displaying the
    mini-window
  . if the computed number of screen lines is more than
    max-mini-window-height allows, then compute where to start the
    mini-window display, as follows:
    - start at the end of the stuff to be displayed in the mini-window
    - move back max-mini-window-height screen lines
    - use the start of the screen line where we wind up as the
      mini-window's start point

IOW, the basic logic is to show the last max-mini-window-height screen
lines of what's in mini-window.

However, when the window-start so computed is then examined by the
code which actually redisplays the mini-window, that code can override
the computed window-start if the position of point will not be visible
in the mini-window with that window-start in effect.  This actually
happens when the test code uses buffer text (not an overlay string) --
the computed window-start, which is in the middle of the "askdjf..."
text, is abandoned, and BOB is used instead.  This does NOT happen
with the overlay-string version, because the window-start point
computed by resize_mini_window is EOB, and that position is visible in
the window.

> >> (and as you know it's wickedly difficult to construct a string which
> >> will have "just the right size" to fit into the minibuffer window).
> > It doesn't have to be "just the right size", it could err on the safe
> > side.  It already attempts to do so, by avoiding truncation in the
> > middle of a candidate.  It should just do a better job, that's all.
> 
> And how do we generalize that to the case where the overlay contains
> newlines, TABs, chars in different scripts using different fonts,
> different faces, images, etc.... ?

Doesn't window-text-pixel-size provide a tool to solve at least some
of those problems?

>     (minibuffer-with-setup-hook
>         (lambda ()
>           (insert "hello")
>           (let ((ol (make-overlay (point) (point)))
>                 (max-mini-window-height 1)
>                 (text "askdjfhaklsjdfhlkasjdfhklasdhflkasdhflkajsdhflkashdfkljahsdlfkjahsdlfkjhasldkfhalskdjfhalskdfhlaksdhfklasdhflkasdhflkasdhflkajsdhklajsdgh"))
>             (save-excursion (insert text))
>             (sit-for 2)
>             (delete-region (point) (point-max))
>             (put-text-property 0 1 'cursor t text)
>             (overlay-put ol 'after-string text)
>             (sit-for 2)
>             (delete-overlay ol)))
>       (read-string "toto: "))

(Btw, people who read this should be aware that binding
max-mini-window-height like this doesn't work in general: the setting
must be in effect when redisplay runs.  It works here only because
there are sit-for calls.)

I believe I explained the issues above; if not, please ask specific
questions.

> So the question is: how to get the same behavior as what we'd get with
> `insert` but without actually modifying the buffer's contents?

You can't, not without redesigning the display code.  At least not in
the general way you describe the issue, and not to the best of my
knowledge.

Without such a redesign we can only make ad-hoc changes for specific
situations.  If such ad-hoc changes are to be done in the display
engine, I need a better, more detailed (and more friendly) spec.

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Eli Zaretskii]

> Date: Sun, 20 Sep 2020 08:27:55 +0000
> From: Gregory Heytings <ghe@sdf.org>
> cc: 43519@debbugs.gnu.org
> 
> But the point is that there is clearly enough horizontal space, so there 
> is (or there does not seem to be) no reason to scroll horizontally.

There's no horizontal scroll.  What happens is that Emacs decides that
the mini-window's display will start at EOB.  I hope I've explained
why this happens in my previous message.

> > You are asking the display engine to do the impossible.
> 
> No.  With a minibuffer-only frame, the bug is not present.

With a minibuffer-only frame, mini-window resizing works very
differently: it resizes the entire frame.

> So it's feasible, and the display engine knows how to do it.

I'm sure we will welcome patches to make this happen in the other
cases.

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Another note: with (setq resize-mini-windows nil) the bug disappears, the 
prompt is always visible even if the overlay text is too large.  So a more 
robust solution to the problem would be to do something like:

(add-hook 'icomplete-minibuffer-setup-hook (function (lambda () (setq resize-mini-windows nil) (enlarge-window N))))

with N replaced by the number of lines that we are about to display.

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Eli Zaretskii]

> Date: Sun, 20 Sep 2020 10:12:14 +0000
> From: Gregory Heytings <ghe@sdf.org>
> cc: 43519@debbugs.gnu.org
> 
> Another note: with (setq resize-mini-windows nil) the bug disappears, the 
> prompt is always visible even if the overlay text is too large.  So a more 
> robust solution to the problem would be to do something like:
> 
> (add-hook 'icomplete-minibuffer-setup-hook (function (lambda () (setq resize-mini-windows nil) (enlarge-window N))))
> 
> with N replaced by the number of lines that we are about to display.

If this is an acceptable solution, it is definitely fine by me.  But I
envision complaints from people who expect certain behavior from
mini-windows with large contents: the above effectively ignores their
settings.

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

After some further analysis, the problem here is in resize_mini_window().

In the comments one finds:

"Set W->start to the right place to begin display.  If the whole contents 
fit, start at the beginning.  Otherwise, start so as to make the end of 
the contents appear.  This is particularly important for y-or-n-p, but 
seems desirable generally."

I won't judge the "seems desirable generally", but in this case at least 
it is clearly not desirable, so there should be a way to do something 
else.

More precisely, in this case height > max_height, so

init_iterator (&it, w, ZV, ZV_BYTE, NULL, DEFAULT_FACE_ID);

is called, followed by

move_it_vertically_backward (&it, height - unit);

which does nothing given that height == unit, so start is set to ZV.

What I would suggest is to add a user option to set start to BEGV when 
height > max_height, which is what is needed here.  It would be reset to 
nil in read_minibuf() before calling minibuffer-setup-hook, and would be 
used in resize_mini_window() as follows:

/* Compute a suitable window start.  */
if (height > max_height && !EQ (Vstart_display_at_beginning_of_minibuffer, Qt))

This would not break any existing behavior.  Would this be a good way to 
solve that problem?

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

For some reason I did not receive a copy of your mail.  I see it now.

>
>  . Fix these problems on the application level.  In the case in
>    point, that would mean for icomplete.el to augment its
>    calculations of where to truncate the list of candidates so that
>    the problem doesn't happen (and AFAICT it doesn't happen if the
>    stuff to be displayed in the mini-window fits the mini-window
>    after resizing it to max-mini-window-height).
>

It is difficult to fix such problems on the application level.  If the 
stuff to be displayed in the mini-window fits the mini-window after 
resizing it to max-mini-window-height, the problem does not happen indeed. 
But the difficulty is precisely to create the stuff to be displayed in 
such a way that it fits the mini-window, because it can use a font that is 
not the default one, it can have embedded newlines, it can contain lines 
that are too wide for the mini-window, and so forth.

>
>  . Define in more detail what situations we would like to fix in the
>    display code, so that we could install special ad-hoc changes
>    there to handle those situations.  For example: is it true that in
>    all of these situations starting the mini-window display at BOB
>    would DTRT?  If so, I think this could be arranged.  If not, why
>    not, and what is the more correct definition of the situations we
>    want to handle?
>

See my proposal, which does exactly this.  Yes, it is true that in all of 
these situations starting the mini-window display at BOB would do the 
right thing.

>
> IOW, the basic logic is to show the last max-mini-window-height screen 
> lines of what's in mini-window.
>

Yes, and this is not desirable in certain cases, it should be possible to 
show the *first* max-mini-window-height screen lines of what's in 
mini-window.

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[Apparently my previous mail was not delivered, so I send it again.  I 
apologize if you receive it twice.]

After some further analysis, the problem here is in resize_mini_window().

In the comments one finds:

"Set W->start to the right place to begin display.  If the whole contents 
fit, start at the beginning.  Otherwise, start so as to make the end of 
the contents appear.  This is particularly important for y-or-n-p, but 
seems desirable generally."

I won't judge the "seems desirable generally", but in this case at least 
it is clearly not desirable, so there should be a way to do something 
else.

More precisely, in this case height > max_height, so

init_iterator (&it, w, ZV, ZV_BYTE, NULL, DEFAULT_FACE_ID);

is called, followed by

move_it_vertically_backward (&it, height - unit);

which does nothing given that height == unit, so start is set to ZV.

What I would suggest is to add a user option to set start to BEGV when 
height > max_height, which is what is needed here.  It would be reset to 
nil in read_minibuf() before calling minibuffer-setup-hook, and would be 
used in resize_mini_window() as follows:

/* Compute a suitable window start.  */
if (height > max_height && !EQ (Vstart_display_at_beginning_of_minibuffer, Qt))

This would not break any existing behavior.  Would this be a good way to 
solve that problem?

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Stefan Monnier]

>> That the redisplay performed horizontal scrolling when it was not needed
>> since the cursor was already visible without such scrolling.
> There's no horizontal scrolling.

To the extent that window-start is not at BOL, I think this qualifies as
horizontal scrolling, but maybe horizontal scrolling has a more specific
meaning within the redisplay with which I'm not aware.

> The issue is with determining the
> mini-window's start position.  In the case with the overlay, we
> compute that position as EOB, whereas in the case with buffer text, we
> compute it to be at BOB.  The reason is what I said: the very
> different behavior of the move_it_* functions when they need to
> traverse overlay strings.
>
> The basic logic of resize_mini_window is like this:
>
>   . compute the number of screen lines required for displaying the
>     mini-window
>   . if the computed number of screen lines is more than
>     max-mini-window-height allows, then compute where to start the
>     mini-window display, as follows:
>     - start at the end of the stuff to be displayed in the mini-window
>     - move back max-mini-window-height screen lines
>     - use the start of the screen line where we wind up as the
>       mini-window's start point

Hmm...  I think I'm beginning to see where the difficulty might be
coming from, but it's still quite fuzzy.  In my mind I'd have expected
the code to work more along the lines of:

    - compute the number of screen lines required for displaying the
      mini-window
    - enlarge/shrink the window accordingly
    - rely on the usual redisplay code for the rest (which may decide to
      change window-start in order to keep point within view, but in
      our current example(s) wouldn't need to do that).

Do you know why we don't do it this way, IOW why don't we first try to
keep window-start unchanged and see if point ends up within view?

>> So the question is: how to get the same behavior as what we'd get with
>> `insert` but without actually modifying the buffer's contents?
> You can't, not without redesigning the display code.  At least not in
> the general way you describe the issue, and not to the best of my
> knowledge.

IIUC the problem only shows up because of the auto-resizing of the
minibuffer window, right?
Indeed if I replace

    (setq max-mini-window-height 1)
with
    (setq resize-mini-windows nil)

the problem doesn't appear, even though resizing is in fact disabled in
both cases.


        Stefan

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>
> What I would suggest is to add a user option to set start to BEGV when 
> height > max_height, which is what is needed here.  It would be reset to 
> nil in read_minibuf() before calling minibuffer-setup-hook, and would be 
> used in resize_mini_window() as follows:
>
> /* Compute a suitable window start.  */
> if (height > max_height && !EQ (Vstart_display_at_beginning_of_minibuffer, Qt))
>
> This would not break any existing behavior.  Would this be a good way to 
> solve that problem?
>

After further testing, doing this is not enough, because it is necessary 
to update height even when start_display_at_beginning_of_minibuffer.  So 
the code should be:

if (height > max_height && !EQ (Vstart_display_at_beginning_of_minibuffer, Qt))
{
   ...
}
else
{
   if (height > max_height) height = (max_height / unit) * unit;
   SET_TEXT_POS (start, BEGV, BEGV_BYTE);
}

Another note to answer Eli's previous questions:

>
>  . Define in more detail what situations we would like to fix in the
>    display code, so that we could install special ad-hoc changes
>    there to handle those situations.  For example: is it true that in
>    all of these situations starting the mini-window display at BOB
>    would DTRT?  If so, I think this could be arranged.  If not, why
>    not, and what is the more correct definition of the situations we
>    want to handle?
>

It is "true that in all of these situations starting the mini-window 
display at BOB would DTRT", but it is also true that in all of these 
situations the height overflow is due to an overlay at EOB.  So another 
solution, which I think would be even better, but might change the 
existing behavior marginally, would be to calculate the height two times, 
once at EOB and once at EOB-1.  With this Emacs would detect by itself the 
situations in which it is better to show the first max-mini-window-height 
lines instead of the last max-mini-window-height lines, and it would not 
be necessary to explicitly set a variable in minibuffer-setup-hook.  The 
code would be:

if (it.line_wrap == TRUNCATE)
   height_near_eob = height = unit;
else
{
   last_height = 0;
   move_it_to (&it, ZV - 1, -1, -1, -1, MOVE_TO_POS);
   if (it.max_ascent == 0 && it.max_descent == 0)
     height_near_eob = it.current_y + last_height;
   else
     height_near_eob = it.current_y + it.max_ascent + it.max_descent;
   height_near_eob -= min (it.extra_line_spacing, it.max_extra_line_spacing);
   move_it_to (&it, ZV, -1, -1, -1, MOVE_TO_POS);
   ...
}

if (height > max_height && height_near_eob > max_height)
{
   ...
}
else
{
   if (height > max_height) height = (max_height / unit) * unit;
   SET_TEXT_POS (start, BEGV, BEGV_BYTE);
}

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>
> IIUC the problem only shows up because of the auto-resizing of the 
> minibuffer window, right?
>

Yes.

>
> Indeed if I replace
>
>    (setq max-mini-window-height 1)
> with
>    (setq resize-mini-windows nil)
>
> the problem doesn't appear, even though resizing is in fact disabled in 
> both cases.
>

Yes, but setting these two variables have a very different effect. 
Setting resize-mini-windows to nil means that resize_mini_window() will 
return immediately after setting the start of display to its default value 
(namely BEGV).  Setting max-mini-window-height to 1 does not have that 
effect, resize_mini_window() computes the height that would be necessary 
to display the content of the minibuffer, compares it to the height that 
is actually available, and uses that height to calculate the start of 
display, by moving backward from the height that would be necessary (which 
does nothing in this case, given that height == unit).

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Eli Zaretskii]

> Date: Sun, 20 Sep 2020 21:04:02 +0000
> From: Gregory Heytings <ghe@sdf.org>
> cc: Stefan Monnier <monnier@iro.umontreal.ca>, 43519@debbugs.gnu.org
> 
> It is difficult to fix such problems on the application level. If the stuff to be displayed in the mini-window fits the mini-window after resizing it to max-mini-window-height, the problem does not happen indeed. But the difficulty is precisely to create the stuff to be displayed in such a way that it fits the mini-window, because it can use a font that is not the default one, it can have embedded newlines, it can contain lines that are too wide for the mini-window, and so forth.

I believe using window-text-pixel-size (which I already mentioned
several times) will avoid any such difficulties, since that function
takes all of those complications into account.  Therefore, I still
don't understand why this approach is not being explored more
actively.

> it is true that in all of these situations starting the mini-window display at BOB would do the right thing.

Are you sure?  What if the prompt is longer than a screen line (i.e.,
the prompt itself is continued on the 2nd and subsequent screen
lines)?  If the prompt is long, but the list of candidates is short,
starting the mini-window display at BOB might fail to show some or all
of the candidates, because the long prompt uses up most or all of the
mini-window screen estate.

>     IOW, the basic logic is to show the last max-mini-window-height screen lines of what's in mini-window.
> 
> Yes, and this is not desirable in certain cases, it should be possible to show the *first* max-mini-window-height screen lines of what's in mini-window. 

Showing the last part is in general a better strategy in the use cases
relevant to the mini-window, which are about user interaction.  I
believe you assume that starting at BOB still shows enough of the text
to allow the user to interact intelligently, but those are not the
only cases we should keep in mind, since the prompt doesn't have to be
short enough for that assumption to be true.

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Eli Zaretskii]

> Date: Sun, 20 Sep 2020 21:31:21 +0000
> From: Gregory Heytings <ghe@sdf.org>
> cc: Stefan Monnier <monnier@iro.umontreal.ca>, 43519@debbugs.gnu.org
> 
> "Set W->start to the right place to begin display. If the whole contents fit, start at the beginning. Otherwise, start so as to make the end of the contents appear. This is particularly important for y-or-n-p, but seems desirable generally."
> 
> I won't judge the "seems desirable generally", but in this case at least it is clearly not desirable, so there should be a way to do something else.

The problem is what is that "something else", exactly, and how to
specify that to redisplay in terms it can understand and follow.

> More precisely, in this case height > max_height, so
> 
> init_iterator (&it, w, ZV, ZV_BYTE, NULL, DEFAULT_FACE_ID);
> 
> is called, followed by
> 
> move_it_vertically_backward (&it, height - unit);
> 
> which does nothing given that height == unit, so start is set to ZV.

height == unit only when max_height = 1.  But the same problem will
happen if max_height = 2 and the stuff we want to display takes more
than 2 screen lines, right?  In those other cases, this code doesn't
"do nothing".

> What I would suggest is to add a user option to set start to BEGV when height > max_height, which is what is needed here. It would be reset to nil in read_minibuf() before calling minibuffer-setup-hook, and would be used in resize_mini_window() as follows:
> 
> /* Compute a suitable window start.  */
> if (height > max_height && !EQ (Vstart_display_at_beginning_of_minibuffer, Qt))

First, I'm not yet convinced starting at BOB is always TRT.  For
example, what if the prompt is very long and takes up more than one
screen line?

Second, it is not enough to set window-start to a particular buffer
position, we must also make sure the position of point will be visible
with that window-start.  Otherwise, redisplay will override the
window-start we set.  So, before setting this flag, the application
will still need some code to see if BOB is pertinent, i.e. consider
the resulting layout, which is something you wanted to avoid in the
first place.

And finally, if a sufficiently generic solution that doesn't require
external knobs can be devised, I'd prefer doing TRT automatically,
without imposing such non-trivial settings on the application.

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: 43519@debbugs.gnu.org
> Date: Sun, 20 Sep 2020 18:40:15 -0400
> 
> > There's no horizontal scrolling.
> 
> To the extent that window-start is not at BOL, I think this qualifies as
> horizontal scrolling, but maybe horizontal scrolling has a more specific
> meaning within the redisplay with which I'm not aware.

We do have horizontal scrolling in Emacs: when lines are truncated.
See auto-hscroll and scroll-left/right.  That's what "horizontal
scrolling" means to me in the context of Emacs.

> Hmm...  I think I'm beginning to see where the difficulty might be
> coming from, but it's still quite fuzzy.  In my mind I'd have expected
> the code to work more along the lines of:
> 
>     - compute the number of screen lines required for displaying the
>       mini-window
>     - enlarge/shrink the window accordingly
>     - rely on the usual redisplay code for the rest (which may decide to
>       change window-start in order to keep point within view, but in
>       our current example(s) wouldn't need to do that).
> 
> Do you know why we don't do it this way, IOW why don't we first try to
> keep window-start unchanged and see if point ends up within view?

Because this way we have no control on where the mini-window's display
will start, and consequently what will be visible in the mini-window.
In particular, if point is at EOB, redisplay could (and normally does)
decide not to position point on the last screen line of the window,
which means we may have some of the text not visible for no good
reason -- not a good thing when user interaction is concerned.

More generally, since the mini-window is usually small and user
interaction requires to make sure the user sees the important parts of
the buffer text (if not all of it), we want tighter control on what
will end up on display, and the way to do that is via setting
window-start.

> IIUC the problem only shows up because of the auto-resizing of the
> minibuffer window, right?
> Indeed if I replace
> 
>     (setq max-mini-window-height 1)
> with
>     (setq resize-mini-windows nil)
> 
> the problem doesn't appear, even though resizing is in fact disabled in
> both cases.

Disabling resize-mini-windows means the mini-window is not resized at
all as part of redisplay.  It doesn't mean the mini-window will always
be a single screen line.  The user or a Lisp program can resize it "by
hand", and it will stay that way until manually resized again.
Basically, disabling this means either the user doesn't care about
what will be visible in the mini-window, or a Lisp program takes the
responsibility of resizing the mini-window as needed.

When resize-mini-windows is disabled, we always start the display at
BOB.  That might be the desired outcome in the particular case you
used as the example, but as I wrote elsewhere, I'm not at all sure
this is TRT for all of the use cases we are discussing: for example,
what if the prompt is longer than a single screen line?  Worse, with
the stuff to display in the mini-window long enough, leaving the
mini-window not resized will obscure large portions of that stuff, so
this cure sounds to me worse than the disease.

And besides, this is a user option, so having some code disregard it
is unfriendly, to say the least, even if we do that temporarily.

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Eli Zaretskii]

> Date: Mon, 21 Sep 2020 06:50:22 +0000
> From: Gregory Heytings <ghe@sdf.org>
> cc: Stefan Monnier <monnier@iro.umontreal.ca>, 43519@debbugs.gnu.org
> 
> It is "true that in all of these situations starting the mini-window display at BOB would DTRT", but it is also true that in all of these situations the height overflow is due to an overlay at EOB.

By "height overflow" you mean the fact that max-mini-window-height
screen lines aren't enough to display all of the stuff we want in the
mini-window?  If so, you are considering only a narrow class of use
cases.  In other use cases, we could have a very long prompt, for
example.  Or we could even have a tall enough image there.  Then
reasons for "overflow" could be different.

> So another solution, which I think would be even better, but might change the existing behavior marginally, would be to calculate the height two times, once at EOB and once at EOB-1.

This assumes that there's no overlay strings at EOB-1?  Why would we
assume something like that?  It again narrows the set of use cases
which will be handled properly.

It also seems to assume that the prompt (which ends at EOB) is much
shorter than the overlay string displayed beyond EOB.  But this is not
guaranteed: it could be the other way around, in which case the
proposed changes will not show the most important part of the
mini-window's display.

My suggestion is to go back to the "start display at BOB" assertion
(which is not true in general), and try to find a more general/more
accurate one.  For example: would it be okay to start the display at
the beginning of the screen line where we end up after
move_it_vertically_backward returns?  This way, if the prompt is very
long, we will start in its middle (at the beginning of one of the
continuation lines used to display the prompt), but we will show as
much of the overlay string as possible.  This strategy will also
handle minibuffer prompts that don't use overlay strings at all better
than if we start at BOB.

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>
> I believe using window-text-pixel-size (which I already mentioned 
> several times) will avoid any such difficulties, since that function 
> takes all of those complications into account.  Therefore, I still don't 
> understand why this approach is not being explored more actively.
>

This I don't know or understand either.  My guess is that creating a 
candidate list by adding one candidate at a time and checking with 
window-text-pixel-size if the result is too large would be inefficient. 
This could be improved with a kind of binary search of course.

>> it is true that in all of these situations starting the mini-window 
>> display at BOB would do the right thing.
>
> Are you sure?  What if the prompt is longer than a screen line (i.e., 
> the prompt itself is continued on the 2nd and subsequent screen lines)? 
> If the prompt is long, but the list of candidates is short, starting the 
> mini-window display at BOB might fail to show some or all of the 
> candidates, because the long prompt uses up most or all of the 
> mini-window screen estate.
>

Yes I'm sure.  In the case you mention indeed some candidates will not be 
displayed, but that's not a problem because most of the time all 
candidates are not displayed anyway.  Of course there is the case when the 
prompt is, say, two characters less than the mini-window width, in which 
case no candidates will be displayed (if the user has (setq 
max-mini-window-width 1)), but this is unlikely to happen, and the default 
value of max-mini-window-width is 0.25 anyway.

>>> IOW, the basic logic is to show the last max-mini-window-height screen 
>>> lines of what's in mini-window.
>>
>> Yes, and this is not desirable in certain cases, it should be possible 
>> to show the *first* max-mini-window-height screen lines of what's in 
>> mini-window.
>
> Showing the last part is in general a better strategy in the use cases 
> relevant to the mini-window, which are about user interaction.
>

In general yes, but not when displaying completion candidates with an 
overlay at EOB, with the point before the overlay text.  In that case you 
start with a blinking cursor at the top left of the minibuffer, without 
any indication of what you are doing or should do.

>
> I believe you assume that starting at BOB still shows enough of the text 
> to allow the user to interact intelligently, but those are not the only 
> cases we should keep in mind, since the prompt doesn't have to be short 
> enough for that assumption to be true.
>

I tested this, and it works, even with overlong prompts.  In that case 
(for example with (setq max-mini-window-width 1) and a prompt wider than 
the mini-window width) the prompt disappears, but this is expected, and 
it's a corner case that almost never happens.  Moreover when displaying 
completion candidates you don't start with a long prompt (except, again, 
in corner cases).

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> More precisely, in this case height > max_height, so
>>
>> init_iterator (&it, w, ZV, ZV_BYTE, NULL, DEFAULT_FACE_ID);
>>
>> is called, followed by
>>
>> move_it_vertically_backward (&it, height - unit);
>>
>> which does nothing given that height == unit, so start is set to ZV.
>
> height == unit only when max_height = 1. But the same problem will 
> happen if max_height = 2 and the stuff we want to display takes more 
> than 2 screen lines, right?  In those other cases, this code doesn't "do 
> nothing".
>

Yes of course, I was commenting under the assumtion that (setq 
max-mini-window-height 1), with which this thread started.  Indeed the 
code does not "do nothing" when max_height is > 1.

>
>> What I would suggest is to add a user option to set start to BEGV when 
>> height > max_height, which is what is needed here. It would be reset to 
>> nil in read_minibuf() before calling minibuffer-setup-hook, and would 
>> be used in resize_mini_window() as follows:
>>
>> /* Compute a suitable window start.  */
>> if (height > max_height && !EQ (Vstart_display_at_beginning_of_minibuffer, Qt))
>
> First, I'm not yet convinced starting at BOB is always TRT.  For 
> example, what if the prompt is very long and takes up more than one 
> screen line?
>

I tested this.  In that case (with max_height = 1) starting at BOB is not 
a problem, because the display engine with override this and display the 
point.  The prompt will be hidden, but this is expected.  It would happen, 
say, when you are completing a file name in a sub-sub-sub-...-directory 
with max_height = 1.

>
> Second, it is not enough to set window-start to a particular buffer 
> position, we must also make sure the position of point will be visible 
> with that window-start.  Otherwise, redisplay will override the 
> window-start we set.  So, before setting this flag, the application will 
> still need some code to see if BOB is pertinent, i.e. consider the 
> resulting layout, which is something you wanted to avoid in the first 
> place.
>

Note that this case (starting display at BOB and having point outside of 
the resulting display area) is highly unlikely.  When it happens, as you 
write, redisplay overrides the window-start that has been set here, and 
it's okay if it does.

>
> And finally, if a sufficiently generic solution that doesn't require 
> external knobs can be devised, I'd prefer doing TRT automatically, 
> without imposing such non-trivial settings on the application.
>

Yes, hence my second proposal, which checks height at EOB and at EOB-1.

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Eli Zaretskii]

> Date: Mon, 21 Sep 2020 14:18:07 +0000
> From: Gregory Heytings <ghe@sdf.org>
> cc: monnier@iro.umontreal.ca, 43519@debbugs.gnu.org
> 
> > I believe using window-text-pixel-size (which I already mentioned 
> > several times) will avoid any such difficulties, since that function 
> > takes all of those complications into account.  Therefore, I still don't 
> > understand why this approach is not being explored more actively.
> 
> This I don't know or understand either.  My guess is that creating a 
> candidate list by adding one candidate at a time and checking with 
> window-text-pixel-size if the result is too large would be inefficient. 

Even if the implementation indeed passes candidates through
window-text-pixel-size one by one, it is not clear to me that the
result will be slow enough to annoy.  We are talking about user
interaction, where speed is not that important.

In any case, I'd want to see numbers before deciding that this is
unacceptable.

> >> it is true that in all of these situations starting the mini-window 
> >> display at BOB would do the right thing.
> >
> > Are you sure?  What if the prompt is longer than a screen line (i.e., 
> > the prompt itself is continued on the 2nd and subsequent screen lines)? 
> > If the prompt is long, but the list of candidates is short, starting the 
> > mini-window display at BOB might fail to show some or all of the 
> > candidates, because the long prompt uses up most or all of the 
> > mini-window screen estate.
> 
> Yes I'm sure.  In the case you mention indeed some candidates will not be 
> displayed, but that's not a problem because most of the time all 
> candidates are not displayed anyway.

But with long enough prompt, you can have _none_ of the candidates
displayed.

> Of course there is the case when the prompt is, say, two characters
> less than the mini-window width, in which case no candidates will be
> displayed (if the user has (setq max-mini-window-width 1)), but this
> is unlikely to happen

"Unlikely to happen" is not a good guideline for making changes in the
display code, IME.  Guess what? most of the things I considered
"unlikely" did happen eventually.

So I prefer a more generally correct solution, especially when it's
not hard to implement.
> > Showing the last part is in general a better strategy in the use cases 
> > relevant to the mini-window, which are about user interaction.
> 
> In general yes, but not when displaying completion candidates with an 
> overlay at EOB, with the point before the overlay text.

I'm not sure the use case with overlays is indeed the indicator that a
different strategy is needed.  but even if it is, the changes you
proposed don't test for the existence of such an overlay, they test
something else, and thus can affect unrelated use cases.

> In that case you start with a blinking cursor at the top left of the
> minibuffer, without any indication of what you are doing or should
> do.

Are you talking about what we see today?  I'm not arguing to leave it
unchanged, I'm talking about what would be the better solution, and
starting always at BOB sounds sub-optimal to me.

> > I believe you assume that starting at BOB still shows enough of the text 
> > to allow the user to interact intelligently, but those are not the only 
> > cases we should keep in mind, since the prompt doesn't have to be short 
> > enough for that assumption to be true.
> 
> I tested this, and it works, even with overlong prompts.  In that case 
> (for example with (setq max-mini-window-width 1) and a prompt wider than 
> the mini-window width) the prompt disappears, but this is expected, and 
> it's a corner case that almost never happens.

The solution I proposed in my other message (assuming that it is
accepted) is more general, I think.  It also covers "corner cases"
which you are willing to disregard.

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Eli Zaretskii]

> Date: Mon, 21 Sep 2020 14:31:49 +0000
> From: Gregory Heytings <ghe@sdf.org>
> cc: monnier@iro.umontreal.ca, 43519@debbugs.gnu.org
> 
> Yes, hence my second proposal, which checks height at EOB and at EOB-1.

That assumes L2R text, btw, among other problems.

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> It is "true that in all of these situations starting the mini-window 
>> display at BOB would DTRT", but it is also true that in all of these 
>> situations the height overflow is due to an overlay at EOB.
>
> By "height overflow" you mean the fact that max-mini-window-height
> screen lines aren't enough to display all of the stuff we want in the
> mini-window?
>

Yes.

>
> If so, you are considering only a narrow class of use cases.  In other 
> use cases, we could have a very long prompt, for example.  Or we could 
> even have a tall enough image there.  Then reasons for "overflow" could 
> be different.
>

I already commented the case of a very long prompt.  For a tall enough 
image, I don't know.

>
>> So another solution, which I think would be even better, but might 
>> change the existing behavior marginally, would be to calculate the 
>> height two times, once at EOB and once at EOB-1.
>
> This assumes that there's no overlay strings at EOB-1?  Why would we 
> assume something like that?  It again narrows the set of use cases which 
> will be handled properly.
>

The problem happens when completion candidates (with icomplete, ido, and I 
guess others are doing the same) are displayed after the point, with an 
overlay at EOB.  If you think this is a too narrow set of use cases, I 
guess the best thing to do is to let applications choose what to do with a 
start_display_at_beginning_of_minibuffer variable, to let applications 
override what "seems desirable generally".  Setting a variable in 
minibuffer-setup-hook is easy.

>
> It also seems to assume that the prompt (which ends at EOB) is much 
> shorter than the overlay string displayed beyond EOB.  But this is not 
> guaranteed: it could be the other way around, in which case the proposed 
> changes will not show the most important part of the mini-window's 
> display.
>

Did you look at what the code currently does?  With emacs -q, M-x 
icomplete-mode, (setq icomplete-separator "\n"), C-x C-f, you have a 
mini-window with a blinking cursor at the top left, and each time you 
enter a directory its name disappears.  This behavior is considered not 
user-friendly, so much that those who implement completion mechanisms use 
complicated workarounds to avoid it.

>
> My suggestion is to go back to the "start display at BOB" assertion 
> (which is not true in general), and try to find a more general/more 
> accurate one.  For example: would it be okay to start the display at the 
> beginning of the screen line where we end up after 
> move_it_vertically_backward returns?
>

IMO, no, this would not be okay, at least not for icomplete/ido/...  What 
users expect is to see the prompt and what they have entered so far until 
they press RET.  Even if that means displaying one less completion 
candidate at the end of the list.  (When use complete a file name or a 
command in a terminal, the prompt and what you have typed so far does not 
disappear.)

>
> This way, if the prompt is very long, we will start in its middle (at 
> the beginning of one of the continuation lines used to display the 
> prompt), but we will show as much of the overlay string as possible. 
> This strategy will also handle minibuffer prompts that don't use overlay 
> strings at all better than if we start at BOB.
>

I see what you mean.  So I propose to simply let applications/users choose 
what they want in each case with a 
start_display_at_beginning_of_minibuffer variable.

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>
> But with long enough prompt, you can have _none_ of the candidates 
> displayed.
>

With a long enough prompt *and* a small enough mini-window, yes, this can 
happen.

>
> "Unlikely to happen" is not a good guideline for making changes in the 
> display code, IME.  Guess what? most of the things I considered 
> "unlikely" did happen eventually.
>

I'm not saying it will not happen, in fact it's easy to make it happen. 
Just create a long enough directory name, (setq max-mini-window-height 1), 
and enter that directory.  I'm just saying that the likelihook it will 
happen is practice is much, much smaller than the other case, where users 
want to see the prompt, what they have typed so far, and some completion 
candidates after the point.

>
> So I prefer a more generally correct solution, especially when it's not 
> hard to implement.
>

I fear it will be hard to find a "more generally correct solution".  In 
fact, it's a correct solution, so you are looking for a "more generally 
_more_ correct solution" ;-)  BTW, the current solution does not claim to 
be a correct solution, but only that it "seems desirable generally".

>> In that case you start with a blinking cursor at the top left of the 
>> minibuffer, without any indication of what you are doing or should do.
>
> Are you talking about what we see today?
>

Yes.

>
> I'm not arguing to leave it unchanged, I'm talking about what would be 
> the better solution, and starting always at BOB sounds sub-optimal to 
> me.
>

I can't think of a better solution.

>
> The solution I proposed in my other message (assuming that it is 
> accepted) is more general, I think.
>

If you mean "to start the display at the beginning of the screen line 
where we end up after move_it_vertically_backward returns", it is IMO 
worse.  At least with the usecase of completion candidates in mind, it is 
better to have one or two less candidates at the bottom of the 
mini-window, and the prompt displayed at the top of the mini-window.

>
> It also covers "corner cases" which you are willing to disregard.
>

I do not disregard them.  I tested them.  The worst that could happen is 
that, in some rare cases, no completion candidates would be displayed in 
the mini-window, in which case the user can hit TAB and they will be 
displayed in a *Completions* buffer.

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Eli Zaretskii]

> Date: Mon, 21 Sep 2020 15:02:31 +0000
> From: Gregory Heytings <ghe@sdf.org>
> cc: monnier@iro.umontreal.ca, 43519@debbugs.gnu.org
> 
> > For example: would it be okay to start the display at the 
> > beginning of the screen line where we end up after 
> > move_it_vertically_backward returns?
> 
> IMO, no, this would not be okay, at least not for icomplete/ido/...

In that case, what I propose will have exactly the same effect as
setting window-start at BOB.  So I don't understand why you say that
this will not do.

Stefan, what do you think about the above proposal?

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Eli Zaretskii]

> Date: Mon, 21 Sep 2020 15:26:25 +0000
> From: Gregory Heytings <ghe@sdf.org>
> cc: monnier@iro.umontreal.ca, 43519@debbugs.gnu.org
> 
> > The solution I proposed in my other message (assuming that it is 
> > accepted) is more general, I think.
> 
> If you mean "to start the display at the beginning of the screen line 
> where we end up after move_it_vertically_backward returns", it is IMO 
> worse.

I think you misunderstand my proposal, because the beginning of the
screen line where we end up in the use case we are discussing is the
BOB.

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>>> For example: would it be okay to start the display at the beginning of 
>>> the screen line where we end up after move_it_vertically_backward 
>>> returns?
>>
>> IMO, no, this would not be okay, at least not for icomplete/ido/...
>
> In that case, what I propose will have exactly the same effect as 
> setting window-start at BOB.  So I don't understand why you say that 
> this will not do.
>

Perhaps I misunderstood something, but for me "start the display at the 
beginning of the screen line where we end up after 
move_it_vertically_backward" would mean that if the prompt and the user 
input so far needs more than one line, only the last line would be 
displayed.  So instead of having, say,

Find file: <user input>
<user input>|
<completion candidates>

(where | represents the cursor) we would only have:

<user input>|
<completion candidates>

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>>> The solution I proposed in my other message (assuming that it is 
>>> accepted) is more general, I think.
>>
>> If you mean "to start the display at the beginning of the screen line 
>> where we end up after move_it_vertically_backward returns", it is IMO 
>> worse.
>
> I think you misunderstand my proposal, because the beginning of the 
> screen line where we end up in the use case we are discussing is the 
> BOB.
>

I'm not sure they are, but I'm not an expert.  If they are, then of course 
your proposal would be okay.

The purpose of the discussion (AFAIU) is to introduce a change to make it 
possible for icomplete/ido/... to put a too long overlay at EOB (with say 
two or three more candidates than can be displayed in the mini-window), in 
such a way that the prompt will always be displayed, and the last part of 
the overlay will not.  What happens with your proposal when 
max-mini-window-height is 10, and the overlay needs 13 lines?  What 
happens when max-mini-window-height, the overlay needs 10 lines, and the 
prompt and user input so far need two lines?

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Eli Zaretskii]

> Date: Mon, 21 Sep 2020 15:44:16 +0000
> From: Gregory Heytings <ghe@sdf.org>
> cc: monnier@iro.umontreal.ca, 43519@debbugs.gnu.org
> 
> Perhaps I misunderstood something, but for me "start the display at the 
> beginning of the screen line where we end up after 
> move_it_vertically_backward" would mean that if the prompt and the user 
> input so far needs more than one line, only the last line would be 
> displayed.  So instead of having, say,
> 
> Find file: <user input>
> <user input>|
> <completion candidates>
> 
> (where | represents the cursor) we would only have:
> 
> <user input>|
> <completion candidates>

Yes.

This bug was filed to request that Emacs behaves with overlay-string in
the minibuffer prompt the same as with regular buffer text.  What I
propose will do that, or as close as possible to it.  By contrast, you
seem to suggest a change in the current behavior for buffer text as
well.  That may or may not be a good idea, but it's a separate issue,
so should be discussed separately (and in that separate discussion I
will generally be opposed to the change you are proposing, because we
had the current behavior for many years, and so changes like the one
you propose run serious risk of breaking expectations of some package
out there).

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> Perhaps I misunderstood something, but for me "start the display at the 
>> beginning of the screen line where we end up after 
>> move_it_vertically_backward" would mean that if the prompt and the user 
>> input so far needs more than one line, only the last line would be 
>> displayed.  So instead of having, say,
>>
>> Find file: <user input>
>> <user input>|
>> <completion candidates>
>>
>> (where | represents the cursor) we would only have:
>>
>> <user input>|
>> <completion candidates>
>
> Yes.
>
> This bug was filed to request that Emacs behaves with overlay-string in 
> the minibuffer prompt the same as with regular buffer text.
>

Hmmm...  no, this bug was filed after a discussion on emacs-devel (about 
implementing vertical icomplete), and the problem is clearly stated by 
Stefan: the prompt and user input so far disappear.

>
> What I propose will do that, or as close as possible to it.  By 
> contrast, you seem to suggest a change in the current behavior for 
> buffer text as well.
>

Which is why my proposal is to not break anything, but only to give 
applications the control of how what they insert in the minibuffer is 
displayed.  A start_display_at_beginning_of_minibuffer variable that would 
be reset in read_minibuf() and that an application could set in 
minibuffer-setup-hook.  I don't understand why you would be opposed to 
such a change.

>
> That may or may not be a good idea, but it's a separate issue, so should 
> be discussed separately
>

It's _not_ a separate issue, it's the issue at hand.

>
> (and in that separate discussion I will generally be opposed to the 
> change you are proposing, because we had the current behavior for many 
> years, and so changes like the one you propose run serious risk of 
> breaking expectations of some package out there).
>

Why would a change that does not change Emacs' behavior in any way except 
if the user requests it "run serious risk of breaking expectations of some 
package out there"?  It only gives application writers the freedom to 
decide what Emacs should do, which is IMO a good thing in general.

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Stefan Monnier]

>> Do you know why we don't do it this way, IOW why don't we first try to
>> keep window-start unchanged and see if point ends up within view?
> Because this way we have no control on where the mini-window's display
> will start, and consequently what will be visible in the mini-window.

Indeed, then we'll just rely on the "generic" behavior, which is
admittedly not focused on single-line (or few lines) windows, but at
least in the current case it works better (and simplifies the code
slightly).

> In particular, if point is at EOB, redisplay could (and normally does)
> decide not to position point on the last screen line of the window,
> which means we may have some of the text not visible for no good
> reason -- not a good thing when user interaction is concerned.

Not sure I understand what you mean.
If point is at EOB, redisplay will make sure EOB is visible.

Or do you mean a situation like:
- minibuffer holds "foo\n"
- the mini window is a single line
- point is at EOB
In which case we'd end up displaying the empty line instead of display "foo"?

AFAICT our ad-hoc scrolling code gives the same result as the generic
scrolling code in that case.

I've been trying out the patch below and haven't bumped into any
surprising behavior yet, but admittedly, I probably lack creativity.

> More generally, since the mini-window is usually small and user
> interaction requires to make sure the user sees the important parts of
> the buffer text (if not all of it), we want tighter control on what
> will end up on display, and the way to do that is via setting
> window-start.

That makes sense in general, but I'm curious to see what are the
concrete cases where our "generic" scrolling logic leads to worse
behavior than the ad-hoc one used here.

So far the patch below fixed the original problem and I haven't been
able to see any regression yet.

>>     (setq max-mini-window-height 1)
>> with
>>     (setq resize-mini-windows nil)
>> the problem doesn't appear, even though resizing is in fact disabled in
>> both cases.
> Disabling resize-mini-windows means the mini-window is not resized at
> all as part of redisplay.

Yes, of course, I was just using it to illustrate what happens when we
don't use the ad-hoc scrolling code from resize_mini_window in the case
where there is in practice no resizing anyway.

> And besides, this is a user option, so having some code disregard it
> is unfriendly, to say the least, even if we do that temporarily.

Oh yes,


        Stefan


diff --git a/src/xdisp.c b/src/xdisp.c
index dfcb1d73e4..b25aa07f1f 100644
--- a/src/xdisp.c
+++ b/src/xdisp.c
@@ -11885,14 +11885,18 @@ resize_mini_window (struct window *w, bool exact_p)
       if (height > max_height)
 	{
 	  height = (max_height / unit) * unit;
-	  init_iterator (&it, w, ZV, ZV_BYTE, NULL, DEFAULT_FACE_ID);
-	  move_it_vertically_backward (&it, height - unit);
-	  start = it.current.pos;
+	  /* bug#43519: Let the redisplay choose the window start!
+           *
+           * init_iterator (&it, w, ZV, ZV_BYTE, NULL, DEFAULT_FACE_ID);
+	   * move_it_vertically_backward (&it, height - unit);
+	   * start = it.current.pos; */
 	}
       else
-	SET_TEXT_POS (start, BEGV, BEGV_BYTE);
+	{
+	  SET_TEXT_POS (start, BEGV, BEGV_BYTE);
 
-      SET_MARKER_FROM_TEXT_POS (w->start, start);
+          SET_MARKER_FROM_TEXT_POS (w->start, start);
+        }
 
       if (EQ (Vresize_mini_windows, Qgrow_only))
 	{

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Stefan Monnier]

Eli Zaretskii [2020-09-21 17:17:23] wrote:
> For example: would it be okay to start the display at
> the beginning of the screen line where we end up after
> move_it_vertically_backward returns?

Sounds reasonable.  Do we have something like tests (or at least a list
of recipes to try) so we can catch regressions?  It seems this part of
the code is trying to satisfy needs which aren't clearly expressed
anywhere, so while this suggestion may sound good to me (just like the
patch I just sent), I'm afraid it will bring back other problems.


        Stefan

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Eli Zaretskii]

> Date: Mon, 21 Sep 2020 16:27:51 +0000
> From: Gregory Heytings <ghe@sdf.org>
> cc: monnier@iro.umontreal.ca, 43519@debbugs.gnu.org
> 
> > This bug was filed to request that Emacs behaves with overlay-string in 
> > the minibuffer prompt the same as with regular buffer text.
> 
> Hmmm...  no, this bug was filed after a discussion on emacs-devel (about 
> implementing vertical icomplete), and the problem is clearly stated by 
> Stefan: the prompt and user input so far disappear.

That's not my reading of the main argument, posted by Stefan in

  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=43519#17

Some quotes:

  I used `icomplete-mode` only as a vehicle to show the underlying
  behavior with a short recipe which exhibits a real-life problem.
  [...]
  This performs "display of text after point" in 2 different ways:
  - first by `insert`.
  - then with an overlay.
  The visual rendering of the text is the same, with the cursor at the
  same place.  When we do it with `insert` there is no horizontal
  scrolling, but when we do it with an overlay the text gets scrolled so
  the cursor is at `window-start`.

  `icomplete` needs the behavior to be the same as with `insert`, but it
  prefers to use an overlay to avoid some undesirable side-effects of
  modifying the actual text.

  So the question is: how to get the same behavior as what we'd get with
  `insert` but without actually modifying the buffer's contents?

My summary of the problem raised by Stefan:

  . icomplete is just an example that exhibits a more general issue
  . the more general issue is that the display after resizing the
    mini-window is different depending on whether it uses plain buffer
    text or an after-string overlay at EOB

If you use Stefan's recipe, but make the prompt much longer, like
this:

    (minibuffer-with-setup-hook
        (lambda ()
          (insert "hello")
          (let ((ol (make-overlay (point) (point)))
                (max-mini-window-height 1)
                (text "askdjfhaklsjdfhlkasjdfhklasdhflkasdhflkajsdhflkashdfkljahsdlfkjahsdlfkjhasldkfhalskdjfhalskdfhlaksdhfklasdhflkasdhflkasdhflkajsdhklajsdgh"))
            (save-excursion (insert text))
            (sit-for 2)
            (delete-region (point) (point-max))
            (put-text-property 0 1 'cursor t text)
            (overlay-put ol 'after-string text)
            (sit-for 2)
            (delete-overlay ol)))
      (read-string "totototototototototototototototototototototototototototototototototototototototototototototototototototo: "))

then in the 1st ("insert text") part of the recipe you will see only
the last part of the prompt.  Which is the long-standing behavior of
resizing mini-windows: if the mini-window cannot be enlarged enough to
show the entire text in it, we show its last part.

Stefan asked for the same behavior when an overlay with after-string
is used.  My suggestion (I hope) will do what he asks, or approximate
that as well as I think is possible under the current design of the
display code.

> Which is why my proposal is to not break anything, but only to give 
> applications the control of how what they insert in the minibuffer is 
> displayed.  A start_display_at_beginning_of_minibuffer variable that would 
> be reset in read_minibuf() and that an application could set in 
> minibuffer-setup-hook.  I don't understand why you would be opposed to 
> such a change.

Because it changes a long-standing behavior with inserting normal text
into the minibuffer.

> > That may or may not be a good idea, but it's a separate issue, so should 
> > be discussed separately
> 
> It's _not_ a separate issue, it's the issue at hand.

I disagree, and I explained above why.

> > (and in that separate discussion I will generally be opposed to the 
> > change you are proposing, because we had the current behavior for many 
> > years, and so changes like the one you propose run serious risk of 
> > breaking expectations of some package out there).
> >
> 
> Why would a change that does not change Emacs' behavior in any way except 
> if the user requests it "run serious risk of breaking expectations of some 
> package out there"?

This is not the user, this is a Lisp program that will do it.  The
behavior will change in that the user will be shown only the first
part of the text, as opposed to the last part we were showing until
now.  Maybe such a change in behavior is desirable (I'm not sure, and
I don't yet have a clear idea how will Lisp programs decide which
behavior to request), but it's a separate issue.

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: 43519@debbugs.gnu.org
> Date: Mon, 21 Sep 2020 13:25:01 -0400
> 
> >> Do you know why we don't do it this way, IOW why don't we first try to
> >> keep window-start unchanged and see if point ends up within view?
> > Because this way we have no control on where the mini-window's display
> > will start, and consequently what will be visible in the mini-window.
> 
> Indeed, then we'll just rely on the "generic" behavior, which is
> admittedly not focused on single-line (or few lines) windows, but at
> least in the current case it works better (and simplifies the code
> slightly).

I believe the "works better" part is only guaranteed if
max-mini-window-height is 1.

> > In particular, if point is at EOB, redisplay could (and normally does)
> > decide not to position point on the last screen line of the window,
> > which means we may have some of the text not visible for no good
> > reason -- not a good thing when user interaction is concerned.
> 
> Not sure I understand what you mean.
> If point is at EOB, redisplay will make sure EOB is visible.
> 
> Or do you mean a situation like:
> - minibuffer holds "foo\n"
> - the mini window is a single line
> - point is at EOB
> In which case we'd end up displaying the empty line instead of display "foo"?

Something like that.  More generally, assume the text to be displayed
in the mini-window is

   111111111111111111
   2222222222222222222222
   33333333333333333333
   444444444444444444444

With point in the 4th line and max-mini-window-height = 4, there's no
guarantee that we will see all the 4 lines, we could see just 3 and an
empty 4th one.  Which means the user is shown only part of the stuff
for no good reason.

> AFAICT our ad-hoc scrolling code gives the same result as the generic
> scrolling code in that case.

Not sure what ad-hoc scrolling code you allude to here.  If you mean
what resize_mini_window does, then it doesn't really scroll, it just
instructs redisplay to use a particular buffer position as the
window-start; if that position makes point visible, redisplay will
comply, and we get what we wanted: we show in the mini-window what we
want to show, instead of leaving that to redisplay's whims.

> I've been trying out the patch below and haven't bumped into any
> surprising behavior yet, but admittedly, I probably lack creativity.

IOW, you leave it entirely to the generic window-display code to
select window-start based just on the value of point?  And only when
the mini-window cannot be enlarged enough?  I wouldn't.

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Gregory Heytings <ghe@sdf.org>,  43519@debbugs.gnu.org
> Date: Mon, 21 Sep 2020 13:28:59 -0400
> 
> Eli Zaretskii [2020-09-21 17:17:23] wrote:
> > For example: would it be okay to start the display at
> > the beginning of the screen line where we end up after
> > move_it_vertically_backward returns?
> 
> Sounds reasonable.

Thanks.  if this is agreed upon, I will try to come up with a (tested)
patch.

> Do we have something like tests (or at least a list of recipes to
> try) so we can catch regressions?

Not for the mini-window display, I'm afraid.

> It seems this part of the code is trying to satisfy needs which
> aren't clearly expressed anywhere, so while this suggestion may
> sound good to me (just like the patch I just sent), I'm afraid it
> will bring back other problems.

That's a risk, yes.  That's why I prefer "small changes", i.e. changes
that aren't supposed to affect scenarios unrelated to the ones
discussed.

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> Which is why my proposal is to not break anything, but only to give 
>> applications the control of how what they insert in the minibuffer is 
>> displayed.  A start_display_at_beginning_of_minibuffer variable that 
>> would be reset in read_minibuf() and that an application could set in 
>> minibuffer-setup-hook.  I don't understand why you would be opposed to 
>> such a change.
>
> Because it changes a long-standing behavior with inserting normal text 
> into the minibuffer.
>

I does not change anything.  Unless the user (in this case, the developer 
of completion applications such as icomplete or ido or ...) chooses to use 
that new possibility.

>
> This is not the user, this is a Lisp program that will do it.  The 
> behavior will change in that the user will be shown only the first part 
> of the text, as opposed to the last part we were showing until now.
>

The behavior will not change unless the developer (who presumably knows 
what they are doing) requests it to change.

>
> Maybe such a change in behavior is desirable (I'm not sure, and I don't 
> yet have a clear idea how will Lisp programs decide which behavior to 
> request), but it's a separate issue.
>

Okay, so shall I file another bug just to have this same discussion again?

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>>> For example: would it be okay to start the display at the beginning of 
>>> the screen line where we end up after move_it_vertically_backward 
>>> returns?
>>
>> Sounds reasonable.
>
> Thanks.  if this is agreed upon, I will try to come up with a (tested) 
> patch.
>

I at least do not agree.  I was the one with whom Stephan was discussing 
when he filed the bug and I gave him the example with which this bug 
report started.  So I think I understand at least what I meant.  Again the 
context is completion mechanisms, in this case icomplete with a vertical 
presentation, but a similar discussion already took place about ido.

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Stefan Monnier]

>> Indeed, then we'll just rely on the "generic" behavior, which is
>> admittedly not focused on single-line (or few lines) windows, but at
>> least in the current case it works better (and simplifies the code
>> slightly).
> I believe the "works better" part is only guaranteed if
> max-mini-window-height is 1.

You're more optimistic than I am: I only said it works better "in the
current case".

> Something like that.  More generally, assume the text to be displayed
> in the mini-window is
>
>    111111111111111111
>    2222222222222222222222
>    33333333333333333333
>    444444444444444444444
>
> With point in the 4th line and max-mini-window-height = 4, there's no
> guarantee that we will see all the 4 lines, we could see just 3 and an
> empty 4th one.

You mean because of something like recentering?  IOW the kind of
situation that (in normal buffer) can (sometimes) be avoided by the
likes of `comint-scroll-show-maximum-output` and
`scroll-conservatively`?

Aah.... OK thanks, so now I did manage to reproduce a test case where
the current behavior is better than with my patch:

Set max-mini-window-height to 4, then do:

    M-: 1 ^Q ^J 2 ^Q ^J 3 ^Q ^J 4 ^Q ^J

With my patch this last C-j causes a recentering, so we can't see line
2 anymore.  Note that this recentering will still happen without my
patch if we add

    M-< M->

since our ad-hoc scrolling only occurs when the buffer's content
is changed.

And of course, we could set `scroll-conservatively` in the (mini)buffers
in which case my patch would thus recover the same behavior as the
current ad-hoc scroll (or even better since now even the `M-< M->`
avoids the recentering).

I'm not sure what would be the best way to "set `scroll-conservatively`
in the (mini)buffers", but the patch below does it "well enough" for my
tests to work.  It's probably not "good enough" for master, OTOH.

>> AFAICT our ad-hoc scrolling code gives the same result as the generic
>> scrolling code in that case.
> Not sure what ad-hoc scrolling code you allude to here.  If you mean
> what resize_mini_window does,

Yes, that's what I called "ad-hoc scroll".

> then it doesn't really scroll, it just instructs redisplay to use
> a particular buffer position as the window-start;

For me, "scrolling" is the act of changing `window-start`, so it does
seem like the right word to describe what this code does.  I called it
"ad-hoc" because it's special cased for that one particular use, as
opposed to the "generic" scroll code which is used in general.

>> I've been trying out the patch below and haven't bumped into any
>> surprising behavior yet, but admittedly, I probably lack creativity.
> IOW, you leave it entirely to the generic window-display code to
> select window-start based just on the value of point?

Yes.  It seems to work very well.
Even the corner case regression above doesn't seem very serious and can
be addressed using the scroll_conservatively code.

> And only when the mini-window cannot be enlarged enough?  I wouldn't.

We could dispense with this special case, but this is the case where
I think it's clearly at least as good a choice as whatever the generic
scrolling will do.


        Stefan


diff --git a/src/xdisp.c b/src/xdisp.c
index dfcb1d73e4..7738ac5380 100644
--- a/src/xdisp.c
+++ b/src/xdisp.c
@@ -11885,14 +11885,18 @@ resize_mini_window (struct window *w, bool exact_p)
       if (height > max_height)
 	{
 	  height = (max_height / unit) * unit;
-	  init_iterator (&it, w, ZV, ZV_BYTE, NULL, DEFAULT_FACE_ID);
-	  move_it_vertically_backward (&it, height - unit);
-	  start = it.current.pos;
+	  /* bug#43519: Let the redisplay choose the window start!
+           *
+           * init_iterator (&it, w, ZV, ZV_BYTE, NULL, DEFAULT_FACE_ID);
+	   * move_it_vertically_backward (&it, height - unit);
+	   * start = it.current.pos; */
 	}
       else
-	SET_TEXT_POS (start, BEGV, BEGV_BYTE);
+	{
+	  SET_TEXT_POS (start, BEGV, BEGV_BYTE);
 
-      SET_MARKER_FROM_TEXT_POS (w->start, start);
+          SET_MARKER_FROM_TEXT_POS (w->start, start);
+        }
 
       if (EQ (Vresize_mini_windows, Qgrow_only))
 	{
@@ -18913,6 +18917,7 @@ redisplay_window (Lisp_Object window, bool just_this_one_p)
 
   /* Try to scroll by specified few lines.  */
   if ((0 < scroll_conservatively
+       || MINI_WINDOW_P (w)
        || 0 < emacs_scroll_step
        || temp_scroll_step
        || NUMBERP (BVAR (current_buffer, scroll_up_aggressively))
@@ -18923,7 +18928,9 @@ redisplay_window (Lisp_Object window, bool just_this_one_p)
       /* The function returns -1 if new fonts were loaded, 1 if
 	 successful, 0 if not successful.  */
       int ss = try_scrolling (window, just_this_one_p,
-			      scroll_conservatively,
+			      (MINI_WINDOW_P (w)
+			       ? SCROLL_LIMIT + 1
+			       : scroll_conservatively),
 			      emacs_scroll_step,
 			      temp_scroll_step, last_line_misfit);
       switch (ss)

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>
> My summary of the problem raised by Stefan:
>
>  . icomplete is just an example that exhibits a more general issue
>  . the more general issue is that the display after resizing the
>    mini-window is different depending on whether it uses plain buffer
>    text or an after-string overlay at EOB
>

I don't understand how you came to understand things in that way, but this 
is neither the meaning of the bug title "Overlay at end of minibuf hides 
minibuf's real content" (real content = non-overlay part), nor what was 
discussed in emacs-devel.  A short summary:

Ergus: [To implement icomplete-vertical] we need to add the exact amount 
of lines as accurate as possible.

Stefan: I *strongly* recommend you design the behavior under the 
assumption that it's OK if there are a few more lines in the (mini)buffer 
than are actually visible.

Me: if there are too many candidates the prompt disappears, leaving the 
cursor at the beginning of the minibuffer, which is counterintuitive.  A 
simple example: after (setq max-mini-window-height 1), with "M-x a" the 
"M-x" prompt and the "a" disappear.

Stefan: That can (and should) be fixed without having to reduce the number 
of candidates inserted in the (mini)buffer.

Ergus: It will be great if you give me an idea about how to do that.

Stefan: You need to figure out why the redisplay decides to hide the 
prompt rather than some other part of the (mini)buffer.

Stefan files this bug.

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[I send this mail again, apparently it was not delivered.  I apologize if 
you receive it twice.]

>> Which is why my proposal is to not break anything, but only to give 
>> applications the control of how what they insert in the minibuffer is 
>> displayed.  A start_display_at_beginning_of_minibuffer variable that 
>> would be reset in read_minibuf() and that an application could set in 
>> minibuffer-setup-hook.  I don't understand why you would be opposed to 
>> such a change.
>
> Because it changes a long-standing behavior with inserting normal text 
> into the minibuffer.
>

I does not change anything.  Unless the user (in this case, the developer 
of completion applications such as icomplete or ido or ...) chooses to use 
that new possibility.

>
> This is not the user, this is a Lisp program that will do it.  The 
> behavior will change in that the user will be shown only the first part 
> of the text, as opposed to the last part we were showing until now.
>

The behavior will not change unless the developer (who presumably knows 
what they are doing) requests it to change.

>
> Maybe such a change in behavior is desirable (I'm not sure, and I don't 
> yet have a clear idea how will Lisp programs decide which behavior to 
> request), but it's a separate issue.
>

Okay, so shall I file another bug just to have this same discussion again?

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[I send this mail again, apparently it was not delivered.  I apologize if 
you receive it twice.]

>>> For example: would it be okay to start the display at the beginning of 
>>> the screen line where we end up after move_it_vertically_backward 
>>> returns?
>>
>> Sounds reasonable.
>
> Thanks.  if this is agreed upon, I will try to come up with a (tested) 
> patch.
>

I at least do not agree.  I was the one with whom Stephan was discussing 
when he filed the bug and I gave him the example with which this bug 
report started.  So I think I understand at least what I meant.  Again the 
context is completion mechanisms, in this case icomplete with a vertical 
presentation, but a similar discussion already took place about ido.

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: 43519@debbugs.gnu.org
> Date: Mon, 21 Sep 2020 15:17:11 -0400
> 
> >> I've been trying out the patch below and haven't bumped into any
> >> surprising behavior yet, but admittedly, I probably lack creativity.
> > IOW, you leave it entirely to the generic window-display code to
> > select window-start based just on the value of point?
> 
> Yes.  It seems to work very well.
> Even the corner case regression above doesn't seem very serious and can
> be addressed using the scroll_conservatively code.

I'd rather we didn't change the behavior all over because of this one
use case.

But if no one is interested in my proposal, you can do whatever you
wish.

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Hi Stefan,

>> IOW, you leave it entirely to the generic window-display code to select 
>> window-start based just on the value of point?
>
> Yes.  It seems to work very well. Even the corner case regression above 
> doesn't seem very serious and can be addressed using the 
> scroll_conservatively code.
>

I tried your patch, and it works very well indeed.  It seems that it's a 
much deeper change to the code than what I suggested, which would have 
given the same effect but only upon request.  Of course I'm fine with this 
too.

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> IOW, you leave it entirely to the generic window-display code to select 
>> window-start based just on the value of point?
>
> Yes.  It seems to work very well. Even the corner case regression above 
> doesn't seem very serious and can be addressed using the 
> scroll_conservatively code.
>

A last note: w->start has already been set to its default value (BEGV) 
just after entering resize_mini_window(), so the else part in your patch 
is not necessary anymore.  Your code has the same effect as simply doing:

if (height > max_height) height = (max_height / unit) * unit;

in place of the "Compute a suitable window start" part.

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Eli Zaretskii]

> Date: Mon, 21 Sep 2020 22:47:40 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: 43519@debbugs.gnu.org
> 
> > From: Stefan Monnier <monnier@iro.umontreal.ca>
> > Cc: 43519@debbugs.gnu.org
> > Date: Mon, 21 Sep 2020 15:17:11 -0400
> > 
> 
> I'd rather we didn't change the behavior all over because of this one
> use case.
> 
> But if no one is interested in my proposal, you can do whatever you
> wish.

Actually, we can do better.  I will soon install a simple change that
fixes the problems discussed in this bug and in all the related
discussions I'm aware of.  We can then take it from there, if some
issues remain.

Thanks.

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[I send this email again, apparently it was not delivered.  I apologize if 
you receive it twice.]

>
> My summary of the problem raised by Stefan:
>
>  . icomplete is just an example that exhibits a more general issue
>  . the more general issue is that the display after resizing the
>    mini-window is different depending on whether it uses plain buffer
>    text or an after-string overlay at EOB
>

I don't understand how you came to understand things in that way, but this 
is neither the meaning of the bug title "Overlay at end of minibuf hides 
minibuf's real content" (real content = non-overlay part), nor what was 
discussed in emacs-devel.  A short summary:

Ergus: [To implement icomplete-vertical] we need to add the exact amount 
of lines as accurate as possible.

Stefan: I *strongly* recommend you design the behavior under the 
assumption that it's OK if there are a few more lines in the (mini)buffer 
than are actually visible.

Me: if there are too many candidates the prompt disappears, leaving the 
cursor at the beginning of the minibuffer, which is counterintuitive.  A 
simple example: after (setq max-mini-window-height 1), with "M-x a" the 
"M-x" prompt and the "a" disappear.

Stefan: That can (and should) be fixed without having to reduce the number 
of candidates inserted in the (mini)buffer.

Ergus: It will be great if you give me an idea about how to do that.

Stefan: You need to figure out why the redisplay decides to hide the 
prompt rather than some other part of the (mini)buffer.

Stefan files this bug.

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[I send this mail again, apparently it was not delivered.  I apologize if 
you receive it twice.]

>> Which is why my proposal is to not break anything, but only to give 
>> applications the control of how what they insert in the minibuffer is 
>> displayed.  A start_display_at_beginning_of_minibuffer variable that 
>> would be reset in read_minibuf() and that an application could set in 
>> minibuffer-setup-hook.  I don't understand why you would be opposed to 
>> such a change.
>
> Because it changes a long-standing behavior with inserting normal text 
> into the minibuffer.
>

I does not change anything.  Unless the user (in this case, the developer 
of completion applications such as icomplete or ido or ...) chooses to use 
that new possibility.

>
> This is not the user, this is a Lisp program that will do it.  The 
> behavior will change in that the user will be shown only the first part 
> of the text, as opposed to the last part we were showing until now.
>

The behavior will not change unless the developer (who presumably knows 
what they are doing) requests it to change.

>
> Maybe such a change in behavior is desirable (I'm not sure, and I don't 
> yet have a clear idea how will Lisp programs decide which behavior to 
> request), but it's a separate issue.
>

Okay, so shall I file another bug just to have this same discussion again?

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[I send this mail again, apparently it was not delivered.  I apologize if 
you receive it twice.]

>>> For example: would it be okay to start the display at the beginning of 
>>> the screen line where we end up after move_it_vertically_backward 
>>> returns?
>>
>> Sounds reasonable.
>
> Thanks.  if this is agreed upon, I will try to come up with a (tested) 
> patch.
>

I at least do not agree.  I was the one with whom Stephan was discussing 
when he filed the bug and I gave him the example with which this bug 
report started.  So I think I understand at least what I meant.  Again the 
context is completion mechanisms, in this case icomplete with a vertical 
presentation, but a similar discussion already took place about ido.

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[I send this mail again, apparently it was not delivered.  I apologize if 
you receive it twice.]

Hi Stefan,

>> IOW, you leave it entirely to the generic window-display code to select 
>> window-start based just on the value of point?
>
> Yes.  It seems to work very well. Even the corner case regression above 
> doesn't seem very serious and can be addressed using the 
> scroll_conservatively code.
>

I tried your patch, and it works very well indeed.  It seems that it's a 
much deeper change to the code than what I suggested, which would have 
given the same effect but only upon request.  Of course I'm fine with this 
too.

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[I send this mail again, apparently it was not delivered.  I apologize if 
you receive it twice.]

>> IOW, you leave it entirely to the generic window-display code to select 
>> window-start based just on the value of point?
>
> Yes.  It seems to work very well. Even the corner case regression above 
> doesn't seem very serious and can be addressed using the 
> scroll_conservatively code.
>

A last note: w->start has already been set to its default value (BEGV) 
just after entering resize_mini_window(), so the else part in your patch 
is not necessary anymore.  Your code has the same effect as simply doing:

if (height > max_height) height = (max_height / unit) * unit;

in place of the "Compute a suitable window start" part.

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Stefan Monnier]

>> I'd rather we didn't change the behavior all over because of this one
>> use case.

I'm not insisting on my solution, if that's what you're worried about.
I'm only working on it to better understand what's at stake.

>> But if no one is interested in my proposal, you can do whatever
>> you wish.

I don't really know how to implement it, so I was waiting for a patch
from you ;-)

> Actually, we can do better.  I will soon install a simple change that
> fixes the problems discussed in this bug and in all the related
> discussions I'm aware of.  We can then take it from there, if some
> issues remain.

Sounds great.  I'll try and see if I can write a test for this problem.


        Stefan

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Eli Zaretskii]

> From: Eli Zaretskii <eliz@gnu.org>
> Date: Tue, 22 Sep 2020 01:26:49 -0400
> Cc: 43519@debbugs.gnu.org
> 
> Actually, we can do better.  I will soon install a simple change that
> fixes the problems discussed in this bug and in all the related
> discussions I'm aware of.  We can then take it from there, if some
> issues remain.

Done.

I will wait for a week or two for reports about things this change
breaks, or about issues still left to be fixed after this commit, and
if nothing surfaces, I will close the bug report.

Thanks.

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Eli Zaretskii]

> Date: Tue, 22 Sep 2020 06:57:59 +0000
> From: Gregory Heytings <ghe@sdf.org>
> cc: monnier@iro.umontreal.ca, 43519@debbugs.gnu.org
> 
> > Maybe such a change in behavior is desirable (I'm not sure, and I don't 
> > yet have a clear idea how will Lisp programs decide which behavior to 
> > request), but it's a separate issue.
> 
> Okay, so shall I file another bug just to have this same discussion again?

Yes, please.

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Eli Zaretskii]

> Date: Tue, 22 Sep 2020 06:58:20 +0000
> From: Gregory Heytings <ghe@sdf.org>
> cc: Stefan Monnier <monnier@iro.umontreal.ca>, 43519@debbugs.gnu.org
> 
> >>> For example: would it be okay to start the display at the beginning of 
> >>> the screen line where we end up after move_it_vertically_backward 
> >>> returns?
> >>
> >> Sounds reasonable.
> >
> > Thanks.  if this is agreed upon, I will try to come up with a (tested) 
> > patch.
> 
> I at least do not agree.

I'm sorry that all my explanations and descriptions of the code didn't
convince you.  However, since this is my area of expertise, and my
responsibility as a co-maintainer of Emacs, I must act in this case
according to my understanding of what is right and what is the most
beneficial for future Emacs development and maintenance.

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[-- Attachment #1: Type: text/plain, Size: 826 bytes --]


>> Actually, we can do better.  I will soon install a simple change that 
>> fixes the problems discussed in this bug and in all the related 
>> discussions I'm aware of.  We can then take it from there, if some 
>> issues remain.
>
> Done.
>
> I will wait for a week or two for reports about things this change 
> breaks, or about issues still left to be fixed after this commit, and if 
> nothing surfaces, I will close the bug report.
>

Alas...

See the two attached screenshots, taken with (setq max-mini-window-height 
1) and icomplete-mode.  "no-completion-candidates" is what a user will see 
with your change, "completion-candidates" is what a user will see without 
your change.

So your change has exactly the problem that you explained was so important 
to avoid, and that I explained was an unlikely corner case.

[-- Attachment #2: Type: image/png, Size: 15593 bytes --]

[-- Attachment #3: Type: image/png, Size: 14851 bytes --]

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>>> Maybe such a change in behavior is desirable (I'm not sure, and I 
>>> don't yet have a clear idea how will Lisp programs decide which 
>>> behavior to request), but it's a separate issue.
>>
>> Okay, so shall I file another bug just to have this same discussion 
>> again?
>
> Yes, please.
>

But you already told me that you will be opposed to such a change...

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Eli Zaretskii]

> Date: Tue, 22 Sep 2020 15:51:22 +0000
> From: Gregory Heytings <ghe@sdf.org>
> cc: monnier@iro.umontreal.ca, 43519@debbugs.gnu.org
> 
> See the two attached screenshots, taken with (setq max-mini-window-height 
> 1) and icomplete-mode.  "no-completion-candidates" is what a user will see 
> with your change, "completion-candidates" is what a user will see without 
> your change.
> 
> So your change has exactly the problem that you explained was so important 
> to avoid, and that I explained was an unlikely corner case.

Thanks.  Please describe the recipe to get this result starting from
"emacs -Q", so that I could look into it.

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Eli Zaretskii]

> Date: Tue, 22 Sep 2020 15:52:14 +0000
> From: Gregory Heytings <ghe@sdf.org>
> cc: monnier@iro.umontreal.ca, 43519@debbugs.gnu.org
> 
> >>> Maybe such a change in behavior is desirable (I'm not sure, and I 
> >>> don't yet have a clear idea how will Lisp programs decide which 
> >>> behavior to request), but it's a separate issue.
> >>
> >> Okay, so shall I file another bug just to have this same discussion 
> >> again?
> >
> > Yes, please.
> >
> 
> But you already told me that you will be opposed to such a change...

Yes, but I can be convinced to change my mind, if there are good
reasons.  Other people might chime in and provide such reasons.

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> See the two attached screenshots, taken with (setq 
>> max-mini-window-height 1) and icomplete-mode. 
>> "no-completion-candidates" is what a user will see with your change, 
>> "completion-candidates" is what a user will see without your change.
>>
>> So your change has exactly the problem that you explained was so 
>> important to avoid, and that I explained was an unlikely corner case.
>
> Thanks.  Please describe the recipe to get this result starting from 
> "emacs -Q", so that I could look into it.
>

Well, it's easy to reproduce, but it depends on the size of the font used 
in the mini-window and the width of your Emacs frame, so I'm not sure it's 
possible to create a step-by-step recipe.  The simplest recipe I can think 
of is:

1. emacs -Q
2. create a long directory name (in your home directory, say)
3. C-x C-f, and enter the name of that directory, but do not press RET
4. narrow the Emacs frame until you see the cursor blinking in the right fringe
5. M-: (setq max-mini-window-height 1)
6. M-x icomplete-mode
7. C-x C-f, and enter the name of the long directory again

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Eli Zaretskii]

> Date: Tue, 22 Sep 2020 16:17:04 +0000
> From: Gregory Heytings <ghe@sdf.org>
> cc: monnier@iro.umontreal.ca, 43519@debbugs.gnu.org
> 
> 1. emacs -Q
> 2. create a long directory name (in your home directory, say)
> 3. C-x C-f, and enter the name of that directory, but do not press RET
> 4. narrow the Emacs frame until you see the cursor blinking in the right fringe
> 5. M-: (setq max-mini-window-height 1)
> 6. M-x icomplete-mode
> 7. C-x C-f, and enter the name of the long directory again

Thanks.

Are you saying that the behavior before the change was better than the
behavior after it?  Before the change you saw some part of the
directory name (not necessarily all of it, depending on how long the
directory name was), and none of the prompt.  After the change you see
the prompt followed by some part of the directory name, which is
smaller than in the previous behavior.

I think the behavior after the change is better: it shows the prompt,
as was required, and it matches the behavior you wanted, which was to
start the display in the mini-window from BOB.

So I don't understand what problems do you see in the new behavior
that wasn't present in the old behavior.

Displaying long stuff in a mini-window that is forced to be small will
always present some problems, no matter what we do.  The best solution
is to fit the stuff to be displayed to the dimensions of the
mini-window, but that is something only a Lisp program can do, the
display engine cannot.

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Eli Zaretskii]

> Date: Tue, 22 Sep 2020 16:17:04 +0000
> From: Gregory Heytings <ghe@sdf.org>
> cc: monnier@iro.umontreal.ca, 43519@debbugs.gnu.org
> 
> 1. emacs -Q
> 2. create a long directory name (in your home directory, say)
> 3. C-x C-f, and enter the name of that directory, but do not press RET
> 4. narrow the Emacs frame until you see the cursor blinking in the right fringe
> 5. M-: (setq max-mini-window-height 1)
> 6. M-x icomplete-mode
> 7. C-x C-f, and enter the name of the long directory again

Thanks.

Are you saying that the behavior before the change was better than the
behavior after it?  Before the change you saw some part of the
directory name (not necessarily all of it, depending on how long the
directory name was), and none of the prompt.  After the change you see
the prompt followed by some part of the directory name (the part shown
is, of course, smaller than in the previous behavior).

I think the behavior after the change is better: it shows the prompt,
as was required, and it matches the behavior you wanted, which was to
start the display in the mini-window from BOB.

So I don't understand what problems do you see in the new behavior
that weren't present in the old behavior.

Displaying long stuff in a mini-window that is forced to be small will
always present some problems, no matter what we do.  The best solution
is to fit the stuff to be displayed to the dimensions of the
mini-window, but that is something only a Lisp program which triggers
the display can do, the display engine cannot.

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Drew Adams]

> Displaying long stuff in a mini-window that is forced to be small will
> always present some problems, no matter what we do.  The best solution
> is to fit the stuff to be displayed to the dimensions of the
> mini-window, but that is something only a Lisp program can do, the
> display engine cannot.

That's true.

One possibility for displaying more than the space
allows is to put the full text on the prompt string
as property `help-echo'.  That works (if someone has a
mouse and thinks to mouseover the prompt), but no, it's
not ideal.  (And it works only when `tooltip-mode' is
enabled.)

Another possibility would be to bind a minibuffer key in
Ido or Icomplete or whatever, to show the full info in a
popup on demand.

No, I'm not proposing anything.  No such hacks sound
good to me.

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>
> Are you saying that the behavior before the change was better than the 
> behavior after it? Before the change you saw some part of the directory 
> name (not necessarily all of it, depending on how long the directory 
> name was), and none of the prompt.  After the change you see the prompt 
> followed by some part of the directory name (the part shown is, of 
> course, smaller than in the previous behavior).
>

I'm not judging your change by my standards, I'm judging it by your 
standards.

You explained that my proposal was unacceptable because a change with 
which no completion candidates are displayed where completion candidates 
would have been displayed without the change, is unacceptable.

>
> Displaying long stuff in a mini-window that is forced to be small will 
> always present some problems, no matter what we do.
>

Indeed, this is exactly what I said, and you replied that I was 
"disregarding" these problems.

>
> The best solution is to fit the stuff to be displayed to the dimensions 
> of the mini-window, but that is something only a Lisp program which 
> triggers the display can do, the display engine cannot.
>

In general I don't know, but for the usecase with which this bug started 
(namely displaying completion candidates after point with an overlay), the 
answer is clearly and definitely no.  The best solution is not to fit the 
stuff to be displayed to the dimensions of the mini-window, the best 
solution is to put a too large overlay at EOB and request that the display 
starts at BOB (and not at BOL as your change does, because this means that 
the prompt and user input so far can disappear, which is 
counter-intuitive).

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>>>>> Maybe such a change in behavior is desirable (I'm not sure, and I 
>>>>> don't yet have a clear idea how will Lisp programs decide which 
>>>>> behavior to request), but it's a separate issue.
>>>>
>>>> Okay, so shall I file another bug just to have this same discussion 
>>>> again?
>>>
>>> Yes, please.
>>>
>>
>> But you already told me that you will be opposed to such a change...
>
> Yes, but I can be convinced to change my mind, if there are good 
> reasons.  Other people might chime in and provide such reasons.
>

Done.  I'm curious to see whether this will lead to something.

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Eli Zaretskii]

> Date: Tue, 22 Sep 2020 20:06:27 +0000
> From: Gregory Heytings <ghe@sdf.org>
> cc: monnier@iro.umontreal.ca, 43519@debbugs.gnu.org
> 
> I'm not judging your change by my standards, I'm judging it by your 
> standards.
> 
> You explained that my proposal was unacceptable because a change with 
> which no completion candidates are displayed where completion candidates 
> would have been displayed without the change, is unacceptable.

This is a misunderstanding: I was talking about the cases where the
text in the mini-window comes from buffer text, not from an overlay.
The simplest example of what I was talking about is what 'message'
does when it shows a very long message, too long to show in the
mini-window in its entirety.  In that case, Emacs after the change
still behaves as I described: it shows the last portion of the text.

But this is impossible to achieve with overlay strings, because Emacs
is unable to start a window's display in the middle of an overlay
string.  It can only start it either at the string's beginning, or
after its end.

> In general I don't know, but for the usecase with which this bug started 
> (namely displaying completion candidates after point with an overlay), the 
> answer is clearly and definitely no.  The best solution is not to fit the 
> stuff to be displayed to the dimensions of the mini-window, the best 
> solution is to put a too large overlay at EOB and request that the display 
> starts at BOB (and not at BOL as your change does, because this means that 
> the prompt and user input so far can disappear, which is 
> counter-intuitive).

In the simplest case, where there's a single overlay string at EOB,
these two are identical.  In a more complex use case, where there are
multiple overlay strings, perhaps interspersed with buffer text, the
behavior now is more in line with the previous one, in that it shows
the last portion of the text.

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> You explained that my proposal was unacceptable because a change with 
>> which no completion candidates are displayed where completion 
>> candidates would have been displayed without the change, is 
>> unacceptable.
>
> This is a misunderstanding: I was talking about the cases where the text 
> in the mini-window comes from buffer text, not from an overlay. The 
> simplest example of what I was talking about is what 'message' does when 
> it shows a very long message, too long to show in the mini-window in its 
> entirety.  In that case, Emacs after the change still behaves as I 
> described: it shows the last portion of the text.
>

I don't understand how you came to think about something like this.  All 
examples I gave (and the example with which this bug thread started) were 
about completion candidates displayed at EOB with an overlay, and I 
mentioned icomplete and ido several times.  I never claimed that in the 
case you mention the display should start at BOB.

>> In general I don't know, but for the usecase with which this bug 
>> started (namely displaying completion candidates after point with an 
>> overlay), the answer is clearly and definitely no.  The best solution 
>> is not to fit the stuff to be displayed to the dimensions of the 
>> mini-window, the best solution is to put a too large overlay at EOB and 
>> request that the display starts at BOB (and not at BOL as your change 
>> does, because this means that the prompt and user input so far can 
>> disappear, which is counter-intuitive).
>
> In the simplest case, where there's a single overlay string at EOB, 
> these two are identical.
>

In the simplest case they are, in more complex cases they are not.  And 
I'm proposing a solution that would work in all cases, not just in the 
simplest ones.

Let's continue this discussion in bug#43572, where I eagerly wait for your 
comments.

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Eli Zaretskii]

> Date: Wed, 23 Sep 2020 07:15:15 +0000
> From: Gregory Heytings <ghe@sdf.org>
> cc: monnier@iro.umontreal.ca, 43519@debbugs.gnu.org
> 
> > This is a misunderstanding: I was talking about the cases where the text 
> > in the mini-window comes from buffer text, not from an overlay. The 
> > simplest example of what I was talking about is what 'message' does when 
> > it shows a very long message, too long to show in the mini-window in its 
> > entirety.  In that case, Emacs after the change still behaves as I 
> > described: it shows the last portion of the text.
> 
> I don't understand how you came to think about something like this.  All 
> examples I gave (and the example with which this bug thread started) were 
> about completion candidates displayed at EOB with an overlay, and I 
> mentioned icomplete and ido several times.  I never claimed that in the 
> case you mention the display should start at BOB.

But the changes you proposed didn't distinguish between the completion
use case and the other use cases.  resize_mini_window is used for any
display of the mini-window` that includes alls displays of the
minibuffer and also the display of the echo area.  The changes you
proposed will affect all of them, because they don't distinguish
between them.

> In the simplest case they are, in more complex cases they are not.  And 
> I'm proposing a solution that would work in all cases, not just in the 
> simplest ones.

The change I installed either improves on the current behavior or
keeps the current behavior in the more complex cases, and improves the
behavior in the simple case that started this bug report.

> Let's continue this discussion in bug#43572, where I eagerly wait for your 
> comments.

Coming up.

bug#43519: 28.0.50; Overlay at end of minibuf hides minibuf's real content

[Stefan Monnier]

>>> IOW, you leave it entirely to the generic window-display code to select
>>> window-start based just on the value of point?
>> Yes.  It seems to work very well. Even the corner case regression above
>> doesn't seem very serious and can be addressed using the
>> scroll_conservatively code.
> A last note: w->start has already been set to its default value (BEGV) just
>   after entering resize_mini_window(), so the else part in your patch is not
>  necessary anymore.  Your code has the same effect as simply doing:
> if (height > max_height) height = (max_height / unit) * unit;
> in place of the "Compute a suitable window start" part.

Indeed, except that I (later) also removed that setting of w->start.


        Stefan


diff --git a/src/xdisp.c b/src/xdisp.c
index c05f97173d..c73212174a 100644
--- a/src/xdisp.c
+++ b/src/xdisp.c
@@ -11799,10 +11799,10 @@ resize_mini_window_1 (void *a1, Lisp_Object exactly)
    means size the window exactly to the size needed.  Otherwise, it's
    only enlarged until W's buffer is empty.
 
-   Set W->start to the right place to begin display.  If the whole
-   contents fit, start at the beginning.  Otherwise, start so as
-   to make the end of the contents appear.  This is particularly
-   important for y-or-n-p, but seems desirable generally.
+   If the whole contents fit, set W->start at the beginning.
+   Otherwise, let redisplay do its thing to make sure point is displayed,
+   so we can control which part is more important by placing point
+   accordingly.
 
    Value is true if the window height has been changed.  */
 
@@ -11824,9 +11824,10 @@ resize_mini_window (struct window *w, bool exact_p)
     return false;
 
   /* By default, start display at the beginning.  */
-  set_marker_both (w->start, w->contents,
-              BUF_BEGV (XBUFFER (w->contents)),
-              BUF_BEGV_BYTE (XBUFFER (w->contents)));
+  /* bug#43519: Let the redisplay choose the window start!
+   * set_marker_both (w->start, w->contents,
+   *            BUF_BEGV (XBUFFER (w->contents)),
+   *            BUF_BEGV_BYTE (XBUFFER (w->contents))); */
 
   /* Nil means don't try to resize.  */
   if ((NILP (Vresize_mini_windows)
@@ -11885,24 +11886,18 @@ resize_mini_window (struct window *w, bool exact_p)
       if (height > max_height)
        {
          height = (max_height / unit) * unit;
-     init_iterator (&it, w, ZV, ZV_BYTE, NULL, DEFAULT_FACE_ID);
-     move_it_vertically_backward (&it, height - unit);
-     /* The following move is usually a no-op when the stuff
-        displayed in the mini-window comes entirely from buffer
-        text, but it is needed when some of it comes from overlay
-        strings, especially when there's an after-string at ZV.
-        This happens with some completion packages, like
-        icomplete, ido-vertical, etc.  With those packages, if we
-        don't force w->start to be at the beginning of a screen
-        line, important parts of the stuff in the mini-window,
-        such as user prompt, will be hidden from view.  */
-     move_it_by_lines (&it, 0);
-     start = it.current.pos;
+   /* bug#43519: Let the redisplay choose the window start!
+           *
+           * init_iterator (&it, w, ZV, ZV_BYTE, NULL, DEFAULT_FACE_ID);
+    * move_it_vertically_backward (&it, height - unit);
+    * start = it.current.pos; */
        }
       else
-   SET_TEXT_POS (start, BEGV, BEGV_BYTE);
+ {
+   SET_TEXT_POS (start, BEGV, BEGV_BYTE);
 
-      SET_MARKER_FROM_TEXT_POS (w->start, start);
+          SET_MARKER_FROM_TEXT_POS (w->start, start);
+        }
 
       if (EQ (Vresize_mini_windows, Qgrow_only))
        {
@@ -18923,6 +18918,7 @@ redisplay_window (Lisp_Object window, bool just_this_one_p)
 
   /* Try to scroll by specified few lines.  */
   if ((0 < scroll_conservatively
+       || MINI_WINDOW_P (w)
        || 0 < emacs_scroll_step
        || temp_scroll_step
        || NUMBERP (BVAR (current_buffer, scroll_up_aggressively))
@@ -18933,7 +18929,9 @@ redisplay_window (Lisp_Object window, bool just_this_one_p)
       /* The function returns -1 if new fonts were loaded, 1 if
         successful, 0 if not successful.  */
       int ss = try_scrolling (window, just_this_one_p,
-                         scroll_conservatively,
+                       (MINI_WINDOW_P (w)
+                        ? SCROLL_LIMIT + 1
+                        : scroll_conservatively),
                              emacs_scroll_step,
                              temp_scroll_step, last_line_misfit);
       switch (ss)