bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Drew Adams]

> I actually thought about putting the text property or overlay on the
> first character of the text inserted into the minibuffer, something
> that can be done in a minibuffer-setup-hook.  This way, it will be
> easy to find.  Of course, having that on any other part of the text
> will not make the job much harder, either.

Caveat: haven't been following this thread.

I put text properties on the first char of the
minibuffer prompt, for some special purposes.
Dunno whether that makes sense in your context.

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[-- Attachment #1: Type: text/plain, Size: 2205 bytes --]


In Emacs 27.1, the mini-window displays the last lines of the minibuffer.

This is, in general, the desired behavior, but in some cases it is not.

One case in which this behavior is not desirable is when completion 
candidates are displayed with an overlay at the end of the buffer.  When 
this overlay is taller than max-mini-window-height, the prompt and the 
user input so far disappear.  A simple example: M-: (setq 
max-mini-window-height 1), M-x icomplete-mode, M-x a.

Because of this behavior, and because it is counter-intuitive / not 
user-friendly when the prompt and user input so far disappear, those who 
create programs that display such completion candidates have been 
struggling to create overlays in such a way that they are (together with 
the prompt and the user input so far) not taller than 
max-mini-window-height.  Doing this is, in general, far from trivial.

The attached patch makes it possible to (selectively) choose to display 
the _first_ lines of the minibuffer instead of its _last_ lines (which is 
and remains the default behavior).  This means that displaying completion 
candidates becomes a trivial task: it suffices to create an overlay with 
completion candidates, without worrying at all about its size (or about 
the size of the prompt and user input), and as many of these candidates as 
possible will automatically be displayed.

For example, implementing vertical icomplete only requires:

(setq icomplete-separator "\n")
(add-hook 'icomplete-minibuffer-setup-hook (lambda () (setq start-display-at-beginning-of-minibuffer t)))

This feature request follows the discussion in bug#43519.  The change 
proposed there by Eli Zaretskii improves the behavior w.r.t. Emacs 27.1, 
but it is still suboptimal to display completion candidates in a 
user-friendly way.  For example:

Find file: <user input>|
<completion candidates>

(where | represents the cursor) will become:

<user input>|
<completion candidates>

when the user input becomes larger than a line.  That is, the "Find file:" 
prompt and the user input on the first line will disappear.

The attached patch does not change the behavior of Emacs in any way, 
unless the feature it introduces is used.

[-- Attachment #2: Type: text/x-diff, Size: 2129 bytes --]

diff --git a/src/minibuf.c b/src/minibuf.c
index f957b2ae17..44852127ed 100644
--- a/src/minibuf.c
+++ b/src/minibuf.c
@@ -657,6 +657,8 @@ read_minibuf (Lisp_Object map, Lisp_Object initial, Lisp_Object prompt,
   if (STRINGP (input_method) && !NILP (Ffboundp (Qactivate_input_method)))
     call1 (Qactivate_input_method, input_method);
 
+  Vstart_display_at_beginning_of_minibuffer = Qnil;
+
   run_hook (Qminibuffer_setup_hook);
 
   /* Don't allow the user to undo past this point.  */
@@ -2097,6 +2099,15 @@ syms_of_minibuf (void)
 uses to hide passwords.  */);
   Vread_hide_char = Qnil;
 
+  DEFVAR_LISP ("start-display-at-beginning-of-minibuffer", Vstart_display_at_beginning_of_minibuffer,
+	       doc: /* Whether to preferably display the beginning of the minibuffer.
+When the mini-window is not large enough to display the complete minibuffer,
+the default behavior is to display the last part of the minibuffer and to
+hide its first part.  This variable reverses that behavior.  This variable
+is reset when the minibuffer is entered, and must be set in
+`minibuffer-setup-hook'.  */);
+  Vstart_display_at_beginning_of_minibuffer = Qnil;
+
   defsubr (&Sactive_minibuffer_window);
   defsubr (&Sset_minibuffer_window);
   defsubr (&Sread_from_minibuffer);
diff --git a/src/xdisp.c b/src/xdisp.c
index 49225c56fe..1b1a34b97f 100644
--- a/src/xdisp.c
+++ b/src/xdisp.c
@@ -11804,7 +11804,7 @@ resize_mini_window (struct window *w, bool exact_p)
 	}
 
       /* Compute a suitable window start.  */
-      if (height > max_height)
+      if (height > max_height && !EQ (Vstart_display_at_beginning_of_minibuffer, Qt))
 	{
 	  height = (max_height / unit) * unit;
 	  init_iterator (&it, w, ZV, ZV_BYTE, NULL, DEFAULT_FACE_ID);
@@ -11812,7 +11812,10 @@ resize_mini_window (struct window *w, bool exact_p)
 	  start = it.current.pos;
 	}
       else
-	SET_TEXT_POS (start, BEGV, BEGV_BYTE);
+	{
+	  if (height > max_height) height = (max_height / unit) * unit;
+	  SET_TEXT_POS (start, BEGV, BEGV_BYTE);
+	}
 
       SET_MARKER_FROM_TEXT_POS (w->start, start);
 

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Eli Zaretskii]

> Date: Tue, 22 Sep 2020 20:57:13 +0000
> From: Gregory Heytings via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> In Emacs 27.1, the mini-window displays the last lines of the minibuffer.
> 
> This is, in general, the desired behavior, but in some cases it is not.
> 
> One case in which this behavior is not desirable is when completion 
> candidates are displayed with an overlay at the end of the buffer.  When 
> this overlay is taller than max-mini-window-height, the prompt and the 
> user input so far disappear.  A simple example: M-: (setq 
> max-mini-window-height 1), M-x icomplete-mode, M-x a.

Actually, on the current master this example does show the "M-x a"
part.

> This feature request follows the discussion in bug#43519.  The change 
> proposed there by Eli Zaretskii improves the behavior w.r.t. Emacs 27.1, 
> but it is still suboptimal to display completion candidates in a 
> user-friendly way.  For example:
> 
> Find file: <user input>|
> <completion candidates>
> 
> (where | represents the cursor) will become:
> 
> <user input>|
> <completion candidates>
> 
> when the user input becomes larger than a line.  That is, the "Find file:" 
> prompt and the user input on the first line will disappear.

I suggest to show a recipe for this, because the few I tried failed to
produce the described effect (with the current master).  Maybe I'm
missing something.

> The attached patch makes it possible to (selectively) choose to display 
> the _first_ lines of the minibuffer instead of its _last_ lines (which is 
> and remains the default behavior).  This means that displaying completion 
> candidates becomes a trivial task: it suffices to create an overlay with 
> completion candidates, without worrying at all about its size (or about 
> the size of the prompt and user input), and as many of these candidates as 
> possible will automatically be displayed.
> 
> For example, implementing vertical icomplete only requires:
> 
> (setq icomplete-separator "\n")
> (add-hook 'icomplete-minibuffer-setup-hook (lambda () (setq start-display-at-beginning-of-minibuffer t)))

I have a couple of comments regarding the proposed change:

 . How will Lisp programs decide when to set this flag and when not to
   set it?  What would be the criteria?  If you are saying that any
   Lisp program that reads from the minibuffer will want that, then
   (assuming that others agree), it would be better to do this
   automatically in the display code.

 . Binding the variable inside the minibuffer-setup-hook will affect
   all the subsequent calls to resize_mini_window, until the next call
   to read-from-minibuffer resets it, which may not be what the Lisp
   program wants, and could have unintended consequences.

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> In Emacs 27.1, the mini-window displays the last lines of the 
>> minibuffer.
>>
>> This is, in general, the desired behavior, but in some cases it is not.
>>
>> One case in which this behavior is not desirable is when completion 
>> candidates are displayed with an overlay at the end of the buffer. 
>> When this overlay is taller than max-mini-window-height, the prompt and 
>> the user input so far disappear.  A simple example: M-: (setq 
>> max-mini-window-height 1), M-x icomplete-mode, M-x a.
>
> Actually, on the current master this example does show the "M-x a" part.
>

Yes, as I explain just below.  It's an improvement that improves most, but 
not all, cases.

>> This feature request follows the discussion in bug#43519.  The change 
>> proposed there by Eli Zaretskii improves the behavior w.r.t. Emacs 
>> 27.1, but it is still suboptimal to display completion candidates in a 
>> user-friendly way.  For example:
>>
>> Find file: <user input>|
>> <completion candidates>
>>
>> (where | represents the cursor) will become:
>>
>> <user input>|
>> <completion candidates>
>>
>> when the user input becomes larger than a line.  That is, the "Find 
>> file:" prompt and the user input on the first line will disappear.
>
> I suggest to show a recipe for this, because the few I tried failed to 
> produce the described effect (with the current master).  Maybe I'm 
> missing something.
>

I just tested this again with current master, and actually the result is 
slighly worse than what I though, when the prompt and user input becomes 
larger than a line you don't see:

<user input>|
<completion candidates>

but:

|<completion candidates>

(IOW the characters at the beginning of the Nth line, with N > 1, 
disappear.)

Again it's difficult to give a simple recipe, because it depends on the 
width of your Emacs frame and of the size of the font used in the 
mini-window.  The simplest recipe I can think of is:

1. create a "long enough" directory name, with say ten subdirectories
2. emacs -Q
3. make the frame width "small enough"
4. M-: (setq max-mini-window-height 5)
5. M-x icomplete-mode
6. M-: (setq icomplete-separator "\n")
7. C-x C-f and enter the "long enough" directory name

What you will see at this point is:

|{<completion candidate 1>
<completion candidate 2>
...
<completion candidate 5>

>
> How will Lisp programs decide when to set this flag and when not to set 
> it?  What would be the criteria?
>

The criteria is simply: should the prompt and user input be displayed? 
IOW, is what Stefan called the "real content" (prompt and user input so 
far) more important that the overlay (which displays completion candidates 
but is merely an unnecessary help for the user)?

Programs such as icomplete and ido, for example, would most likely want to 
set this flag.

(A more precise way to formulate that criteria would be: should the prompt 
and user input be displayed, unless it is impossible to display them?)

>
> If you are saying that any Lisp program that reads from the minibuffer 
> will want that, then (assuming that others agree), it would be better to 
> do this automatically in the display code.
>

This is not what I'm saying, and I would not dare to make such a general 
judgment.  I only claim that it is better to make this possible.

There is at least one case where I think it is better not to do this 
automatically.  As Stefan indicated in bug#43519, with M-:, when you input 
data, the current behavior is to always have point on the last line of the 
minibuffer.  Doing this automatically (that is, unconditionally) would 
have the consequence that when point reaches the last line of the 
minibuffer (that is, the max-mini-window-height's line), the mini-buffer 
would be recentered, and the topmost lines would be hidden.  This happens 
because the default value of scroll-conservatively is 0; when it is set to 
101 it does not happen anymore.

This is just one case, there are possibly many other cases.  But IMO, the 
mini-buffer is so central to Emacs, and the current behavior is so old 
(twenty years), that I believe changing it requires a lot of care.  This 
could be done in small steps:

1. first with this patch (or if you want with the opposite patch: a 
variable start-display-at-end-of-minibuffer reset to t whenever the 
mini-buffer is entered), which would make it possible to everyone to try 
to set that variable to its non-default value to see if undesirable 
behaviors arise,

2. then by changing the default value to its opposite, say in Emacs 29, if 
it became clear enough that the new behavior does not give rise to any 
undesiable consequences,

3. and finally, in Emacs 3X, by removing that variable.

>
> Binding the variable inside the minibuffer-setup-hook will affect all 
> the subsequent calls to resize_mini_window, until the next call to 
> read-from-minibuffer resets it, which may not be what the Lisp program 
> wants, and could have unintended consequences.
>

I can't think of such unintended consequences.  In the use case of 
displaying completion candidates, this (the fact that it affects all 
successive calls to resize_mini_window) is indeed what is wanted.

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Eli Zaretskii]

> Date: Wed, 23 Sep 2020 19:15:40 +0000
> From: Gregory Heytings <ghe@sdf.org>
> cc: 43572@debbugs.gnu.org
> 
> I just tested this again with current master, and actually the result is 
> slighly worse than what I though, when the prompt and user input becomes 
> larger than a line you don't see:
> 
> <user input>|
> <completion candidates>
> 
> but:
> 
> |<completion candidates>
> 
> (IOW the characters at the beginning of the Nth line, with N > 1, 
> disappear.)
> 
> Again it's difficult to give a simple recipe, because it depends on the 
> width of your Emacs frame and of the size of the font used in the 
> mini-window.  The simplest recipe I can think of is:
> 
> 1. create a "long enough" directory name, with say ten subdirectories
> 2. emacs -Q
> 3. make the frame width "small enough"
> 4. M-: (setq max-mini-window-height 5)
> 5. M-x icomplete-mode
> 6. M-: (setq icomplete-separator "\n")
> 7. C-x C-f and enter the "long enough" directory name
> 
> What you will see at this point is:
> 
> |{<completion candidate 1>
> <completion candidate 2>
> ...
> <completion candidate 5>

Is this worse than before the change?

And given the policy of displaying the last visible part, what would
you expect in this case?

> > How will Lisp programs decide when to set this flag and when not to set 
> > it?  What would be the criteria?
> 
> The criteria is simply: should the prompt and user input be displayed? 

How do you decide that?  Or let me ask it differently: when will a
program decide that it wants the current behavior of perhaps NOT
showing the prompt, if the mini-window is not large enough?

> (A more precise way to formulate that criteria would be: should the prompt 
> and user input be displayed, unless it is impossible to display them?)

We are discussing the case when it's impossible to display both; the
case when it's possible is easy and is already handled.

> There is at least one case where I think it is better not to do this 
> automatically.  As Stefan indicated in bug#43519, with M-:, when you input 
> data, the current behavior is to always have point on the last line of the 
> minibuffer.

That case doesn't need any special handling in resize_mini_window,
because the display engine will always make sure point is visible.  If
the window-start point determined by resize_mini_window doesn't allow
point to be visible, the display engine will find another
window-start, which would.

> Doing this automatically (that is, unconditionally) would 
> have the consequence that when point reaches the last line of the 
> minibuffer (that is, the max-mini-window-height's line), the mini-buffer 
> would be recentered, and the topmost lines would be hidden.

What resize_mini_window does ensures that recentering doesn't happen.
That is why it sets w->start: it's an indication to the display engine
to obey that window-start position if point is visible with it.

So you are trying to solve a case that doesn't need to be solved.

> > Binding the variable inside the minibuffer-setup-hook will affect all 
> > the subsequent calls to resize_mini_window, until the next call to 
> > read-from-minibuffer resets it, which may not be what the Lisp program 
> > wants, and could have unintended consequences.
> 
> I can't think of such unintended consequences.  In the use case of 
> displaying completion candidates, this (the fact that it affects all 
> successive calls to resize_mini_window) is indeed what is wanted.

Well, I _can_ think of such consequences.  As I said,
resize_mini_window is called in many situations that don't involve
completion, so setting that variable to affect all of them is a bad
idea.  We need something more fine-grained if we want to implement
such a feature.

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>
> And given the policy of displaying the last visible part, what would you 
> expect in this case?
>

I do not expect something else with the current policy, I suggest to add a 
new policy with whom the first part of the minibuffer would be displayed 
instead of its last part.

>>> How will Lisp programs decide when to set this flag and when not to 
>>> set it?  What would be the criteria?
>>
>> The criteria is simply: should the prompt and user input be displayed?
>
> How do you decide that?
>

I gave a simple and clear example of an implementation of vertical 
icomplete, which demonstrates how this feature would be used:

(setq icomplete-separator "\n")
(add-hook 'icomplete-minibuffer-setup-hook (lambda () (setq start-display-at-beginning-of-minibuffer t)))

Please try it (together with my patch), you'll see what I mean.

With the current behavior (with (setq icomplete-separator "\n")), whenever 
the prompt and user input are larger than the width of Emacs' frame and 
there are too many completion candidates, the prompt and user input 
disappear, and when there are only a few completion candidates the prompt 
and user input re-appear.

With this patch the prompt and user input are always visible, and as many 
completion candidates as possible (given max-mini-window-height) are 
displayed after them.

>
> Or let me ask it differently: when will a program decide that it wants 
> the current behavior of perhaps NOT showing the prompt, if the 
> mini-window is not large enough?
>

In the use case of completion candidates displayed after point with an 
overlay at EOB, the answer is simple: never.  If at some point it becomes 
impossible to display the prompt, say because max-mini-window-height 
equals 1 and the prompt and user input are larger than the width of the 
Emacs frame, redisplay will hide the prompt, and that's fine.  As you 
said: "Displaying long stuff in a mini-window that is forced to be small 
will always present some problems, no matter what we do."

>> Doing this automatically (that is, unconditionally) would have the 
>> consequence that when point reaches the last line of the minibuffer 
>> (that is, the max-mini-window-height's line), the mini-buffer would be 
>> recentered, and the topmost lines would be hidden.
>
> What resize_mini_window does ensures that recentering doesn't happen. 
> That is why it sets w->start: it's an indication to the display engine 
> to obey that window-start position if point is visible with it.
>
> So you are trying to solve a case that doesn't need to be solved.
>

I'm not trying to solve any problem here.  You said: "If you are saying 
that any Lisp program that reads from the minibuffer will want that, then 
(assuming that others agree), it would be better to do this automatically 
in the display code."  I answered this by saying: "This is not what I'm 
saying, and I would not dare to make such a general judgment.  I only 
claim that it is better to make this possible.  There is at least one case 
where I think it is better not to do this automatically."  And I explained 
that case.

>>> Binding the variable inside the minibuffer-setup-hook will affect all 
>>> the subsequent calls to resize_mini_window, until the next call to 
>>> read-from-minibuffer resets it, which may not be what the Lisp program 
>>> wants, and could have unintended consequences.
>>
>> I can't think of such unintended consequences.  In the use case of 
>> displaying completion candidates, this (the fact that it affects all 
>> successive calls to resize_mini_window) is indeed what is wanted.
>
> Well, I _can_ think of such consequences.  As I said, resize_mini_window 
> is called in many situations that don't involve completion, so setting 
> that variable to affect all of them is a bad idea.  We need something 
> more fine-grained if we want to implement such a feature.
>

It would be very helpful if you could clarify what the consequences you 
think of are.

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Eli Zaretskii]

> Date: Wed, 23 Sep 2020 20:15:54 +0000
> From: Gregory Heytings <ghe@sdf.org>
> cc: 43572@debbugs.gnu.org
> 
> >> I can't think of such unintended consequences.  In the use case of 
> >> displaying completion candidates, this (the fact that it affects all 
> >> successive calls to resize_mini_window) is indeed what is wanted.
> >
> > Well, I _can_ think of such consequences.  As I said, resize_mini_window 
> > is called in many situations that don't involve completion, so setting 
> > that variable to affect all of them is a bad idea.  We need something 
> > more fine-grained if we want to implement such a feature.
> >
> 
> It would be very helpful if you could clarify what the consequences you 
> think of are.

Just grep the sources for the callers of resize_mini_window, and you
will see what I mean.  It is not safe to avoid affecting them using
the techniques you propose.  The text itself should request such
special treatment.

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>>>> I can't think of such unintended consequences.  In the use case of 
>>>> displaying completion candidates, this (the fact that it affects all 
>>>> successive calls to resize_mini_window) is indeed what is wanted.
>>>
>>> Well, I _can_ think of such consequences.  As I said, 
>>> resize_mini_window is called in many situations that don't involve 
>>> completion, so setting that variable to affect all of them is a bad 
>>> idea.  We need something more fine-grained if we want to implement 
>>> such a feature.
>>>
>>
>> It would be very helpful if you could clarify what the consequences you 
>> think of are.
>
> Just grep the sources for the callers of resize_mini_window, and you 
> will see what I mean.
>

I did this.  There are only seven calls to resize_mini_window().

>
> It is not safe to avoid affecting them using the techniques you propose.
>

Could you please provide a recipe which would demonstrate a problem with 
the technique I propose?  AFAICS, the flag is reset immediately when 
read_minibuf() / read-from-minibuffer has ended.

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Eli Zaretskii]

> Date: Thu, 24 Sep 2020 14:41:46 +0000
> From: Gregory Heytings <ghe@sdf.org>
> cc: 43572@debbugs.gnu.org
> 
> > It is not safe to avoid affecting them using the techniques you propose.
> >
> 
> Could you please provide a recipe which would demonstrate a problem with 
> the technique I propose?  AFAICS, the flag is reset immediately when 
> read_minibuf() / read-from-minibuffer has ended.

That is true, but read_minibuf enters recursive-edit, and while that
is active, any call to resize_mini_window will be affected.

Using a text property for this is easy and mostly boilerplate, and it
completely avoids this issue.

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> Could you please provide a recipe which would demonstrate a problem 
>> with the technique I propose?  AFAICS, the flag is reset immediately 
>> when read_minibuf() / read-from-minibuffer has ended.
>
> That is true, but read_minibuf enters recursive-edit, and while that is 
> active, any call to resize_mini_window will be affected.
>

This seems to be a really minor problem.  What kind of problems could 
happen because during a recursive edit while icomplete or ido or ... is 
active, redisplay is asked to start displaying at BOB?  Do you really 
believe this could lead to a non-minor problem?

>
> Using a text property for this is easy and mostly boilerplate, and it 
> completely avoids this issue.
>

Then I wonder why this did not happen earlier (and wonder when it will 
happen).  A bug report about this exact problem has already been filed 
five years ago (bug#24293).  Five years ago.  And the patch I'm proposing 
is less than ten lines long, and would be most welcome today.

>
> I already explained why your design is problematic.  And so did Stefan.
>

No, Stefan helped me to improve my design.

>
> So I don't see why a recipe would be needed.
>

Indeed, if the only remaining problem is that during a recursive edit 
while icomplete or ido or ... is active, redisplay is asked to start 
displaying at BOB instead of BOL, I don't think such a recipe would be 
convincing.

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Eli Zaretskii]

> Date: Thu, 24 Sep 2020 16:09:25 +0000
> From: Gregory Heytings <ghe@sdf.org>
> cc: 43572@debbugs.gnu.org
> 
> > That is true, but read_minibuf enters recursive-edit, and while that is 
> > active, any call to resize_mini_window will be affected.
> 
> This seems to be a really minor problem.  What kind of problems could 
> happen because during a recursive edit while icomplete or ido or ... is 
> active, redisplay is asked to start displaying at BOB?  Do you really 
> believe this could lead to a non-minor problem?

Yes.  There are third-party packages out there that do unimaginable
things when the user is prompted to enter his/her choice of something.

And in any case, we don't want to rely on luck in these matters, IME
the Murphy laws are very strong here.

> > Using a text property for this is easy and mostly boilerplate, and it 
> > completely avoids this issue.
> 
> Then I wonder why this did not happen earlier (and wonder when it will 
> happen).  A bug report about this exact problem has already been filed 
> five years ago (bug#24293).  Five years ago.  And the patch I'm proposing 
> is less than ten lines long, and would be most welcome today.

It didn't happen for the usual reasons: no one felt motivated enough
to sit down and do it.

Can I convince you to try implementing this via text properties?  The
code will be very simple, I promise.  And we are here to help if you
are unsure how to go about that.

> > I already explained why your design is problematic.  And so did Stefan.
> 
> No, Stefan helped me to improve my design.

He did both.

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> This seems to be a really minor problem.  What kind of problems could 
>> happen because during a recursive edit while icomplete or ido or ... is 
>> active, redisplay is asked to start displaying at BOB?  Do you really 
>> believe this could lead to a non-minor problem?
>
> Yes.  There are third-party packages out there that do unimaginable 
> things when the user is prompted to enter his/her choice of something.
>
> And in any case, we don't want to rely on luck in these matters, IME the 
> Murphy laws are very strong here.
>

I can't believe this, in particular given that the default value of 
enable-recursive-minibuffers is nil.  And in any case I'd say that it's to 
the third-party packages to adapt to changes in Emacs (and I'm sure from 
time to time there are changes here and there that force them to adapt 
their code).

>> Then I wonder why this did not happen earlier (and wonder when it will 
>> happen).  A bug report about this exact problem has already been filed 
>> five years ago (bug#24293).  Five years ago.  And the patch I'm 
>> proposing is less than ten lines long, and would be most welcome today.
>
> It didn't happen for the usual reasons: no one felt motivated enough to 
> sit down and do it.
>
> Can I convince you to try implementing this via text properties?
>

Of course you can.  But only if you promise you will not reject what I do.

>
> The code will be very simple, I promise.
>

In fact, that's not motivating ;-)

>
> And we are here to help if you are unsure how to go about that.
>

Thank you, that's good to know.

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Eli Zaretskii]

> Date: Thu, 24 Sep 2020 16:40:58 +0000
> From: Gregory Heytings <ghe@sdf.org>
> cc: 43572@debbugs.gnu.org
> 
> > And in any case, we don't want to rely on luck in these matters, IME the 
> > Murphy laws are very strong here.
> 
> I can't believe this, in particular given that the default value of 
> enable-recursive-minibuffers is nil.

You again are thinking only about uses of mini-window for editing in
the minibuffer.  But that is not the only use of the mini-window.

And even in its use for the minibuffer, many users enable recursive
minibuffers.  I would not be surprised if some specialized modes and
packages enabled it for their operations.

> And in any case I'd say that it's to the third-party packages to
> adapt to changes in Emacs

If we break too many important packages, we will have bug reports
which we will be unable to ignore.  Emacs is expected to not break
third-party code too much too hard.

> > Can I convince you to try implementing this via text properties?
> 
> Of course you can.  But only if you promise you will not reject what I do.

If the implementation is clean, and doesn't present backward
compatibility problems, why would I want to reject it?

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[-- Attachment #1: Type: text/plain, Size: 1213 bytes --]


>> I can't believe this, in particular given that the default value of 
>> enable-recursive-minibuffers is nil.
>
> You again are thinking only about uses of mini-window for editing in the 
> minibuffer.  But that is not the only use of the mini-window.
>

I'm not sure I understand what you mean, but it seems to me that these 
other uses of the mini-window are not at all affected by the proposed 
patch, given that `start-display-at-beginning-of-minibuffer' is reset 
immediately when read_minibuf() / read-from-minibuffer has ended.

>
> And even in its use for the minibuffer, many users enable recursive 
> minibuffers.  I would not be surprised if some specialized modes and 
> packages enabled it for their operations.
>

If this case is important, the attached corrected patch also disables 
setting `start-display-at-beginning-of-minibuffer' in recursive 
minibuffers, that is, it limits the effect of that variable to 
non-recursive minibuffers.

I'll wait until you and Stefan agree on the way to solve that problem in a 
better way to start working on this.  In his last mail he is apparently 
not sure anymore that using text properties to do this, as he suggested 
yesterday, is the best solution.

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-diff; name=start-display-at-beginning-of-minibuffer.patch, Size: 2948 bytes --]

diff --git a/src/minibuf.c b/src/minibuf.c
index f957b2ae17..75ab82278b 100644
--- a/src/minibuf.c
+++ b/src/minibuf.c
@@ -472,6 +472,8 @@ read_minibuf (Lisp_Object map, Lisp_Object initial, Lisp_Object prompt,
     = Fcons (Voverriding_local_map,
 	     Fcons (minibuf_window,
 		    minibuf_save_list));
+  minibuf_save_list
+    = Fcons (Vstart_display_at_beginning_of_minibuffer, minibuf_save_list);
   minibuf_save_list
     = Fcons (minibuf_prompt,
 	     Fcons (make_fixnum (minibuf_prompt_width),
@@ -659,6 +661,8 @@ read_minibuf (Lisp_Object map, Lisp_Object initial, Lisp_Object prompt,
 
   run_hook (Qminibuffer_setup_hook);
 
+  if (minibuf_level > 1) Vstart_display_at_beginning_of_minibuffer = Qnil;
+
   /* Don't allow the user to undo past this point.  */
   bset_undo_list (current_buffer, Qnil);
 
@@ -802,6 +806,8 @@ read_minibuf_unwind (void)
   minibuf_save_list = Fcdr (minibuf_save_list);
   Vminibuffer_history_variable = Fcar (minibuf_save_list);
   minibuf_save_list = Fcdr (minibuf_save_list);
+  Vstart_display_at_beginning_of_minibuffer = Fcar (minibuf_save_list);
+  minibuf_save_list = Fcdr (minibuf_save_list);
   Voverriding_local_map = Fcar (minibuf_save_list);
   minibuf_save_list = Fcdr (minibuf_save_list);
 #if 0
@@ -2097,6 +2103,15 @@ syms_of_minibuf (void)
 uses to hide passwords.  */);
   Vread_hide_char = Qnil;
 
+  DEFVAR_LISP ("start-display-at-beginning-of-minibuffer", Vstart_display_at_beginning_of_minibuffer,
+	       doc: /* Whether to preferably display the beginning of the minibuffer.
+When the mini-window is not large enough to display the complete minibuffer,
+the default behavior is to display the last part of the minibuffer and to
+hide its first part.  This variable reverses that behavior when it is set
+in `minibuffer-setup-hook'.  It is reset before and after each read from
+the minibuffer, and unset in recursive minibuffers.  */);
+  Vstart_display_at_beginning_of_minibuffer = Qnil;
+
   defsubr (&Sactive_minibuffer_window);
   defsubr (&Sset_minibuffer_window);
   defsubr (&Sread_from_minibuffer);
diff --git a/src/xdisp.c b/src/xdisp.c
index 49225c56fe..1b1a34b97f 100644
--- a/src/xdisp.c
+++ b/src/xdisp.c
@@ -11804,7 +11804,7 @@ resize_mini_window (struct window *w, bool exact_p)
 	}
 
       /* Compute a suitable window start.  */
-      if (height > max_height)
+      if (height > max_height && !EQ (Vstart_display_at_beginning_of_minibuffer, Qt))
 	{
 	  height = (max_height / unit) * unit;
 	  init_iterator (&it, w, ZV, ZV_BYTE, NULL, DEFAULT_FACE_ID);
@@ -11812,7 +11812,10 @@ resize_mini_window (struct window *w, bool exact_p)
 	  start = it.current.pos;
 	}
       else
-	SET_TEXT_POS (start, BEGV, BEGV_BYTE);
+	{
+	  if (height > max_height) height = (max_height / unit) * unit;
+	  SET_TEXT_POS (start, BEGV, BEGV_BYTE);
+	}
 
       SET_MARKER_FROM_TEXT_POS (w->start, start);
 

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Eli Zaretskii]

> Date: Thu, 24 Sep 2020 21:51:43 +0000
> From: Gregory Heytings <ghe@sdf.org>
> cc: 43572@debbugs.gnu.org
> 
> >> I can't believe this, in particular given that the default value of 
> >> enable-recursive-minibuffers is nil.
> >
> > You again are thinking only about uses of mini-window for editing in the 
> > minibuffer.  But that is not the only use of the mini-window.
> 
> I'm not sure I understand what you mean, but it seems to me that these 
> other uses of the mini-window are not at all affected by the proposed 
> patch, given that `start-display-at-beginning-of-minibuffer' is reset 
> immediately when read_minibuf() / read-from-minibuffer has ended.

I remind you the fact that read_minibuf enters recursive-edit, during
which any of the other callers of resize_mini_window can be called.

> > And even in its use for the minibuffer, many users enable recursive 
> > minibuffers.  I would not be surprised if some specialized modes and 
> > packages enabled it for their operations.
> 
> If this case is important, the attached corrected patch also disables 
> setting `start-display-at-beginning-of-minibuffer' in recursive 
> minibuffers, that is, it limits the effect of that variable to 
> non-recursive minibuffers.

That's a limitation I'd prefer not to impose.  It also doesn't affect
the other callers of resize_mini_window.

> I'll wait until you and Stefan agree on the way to solve that problem in a 
> better way to start working on this.  In his last mail he is apparently 
> not sure anymore that using text properties to do this, as he suggested 
> yesterday, is the best solution.

Yes, I'm still unsure why Stefan said that, and am waiting for his
elaborations.

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> I'm not sure I understand what you mean, but it seems to me that these 
>> other uses of the mini-window are not at all affected by the proposed 
>> patch, given that `start-display-at-beginning-of-minibuffer' is reset 
>> immediately when read_minibuf() / read-from-minibuffer has ended.
>
> I remind you the fact that read_minibuf enters recursive-edit, during 
> which any of the other callers of resize_mini_window can be called.
>

Now I think I understand what you mean.  Mini-windows are used for 
minibuffers and for the echo area.  So for example, when 
`start-display-at-beginning-of-minibuffer' is set while icomplete is 
active in a frame, a call to `message' in another frame will display the 
first part of the string instead of its last part if the string is too 
large.  This seems like a really minor problem.

>>> And even in its use for the minibuffer, many users enable recursive 
>>> minibuffers.  I would not be surprised if some specialized modes and 
>>> packages enabled it for their operations.
>>
>> If this case is important, the attached corrected patch also disables 
>> setting `start-display-at-beginning-of-minibuffer' in recursive 
>> minibuffers, that is, it limits the effect of that variable to 
>> non-recursive minibuffers.
>
> That's a limitation I'd prefer not to impose.
>

I would also prefer not to impose that limitation, I added it because you 
asked it (or at least that's what I understood).  I think the second patch 
I sent is the best and simplest one.  I do agree with you that it's an 
ad-hoc solution, but there are other such ad-hoc solutions in Emacs, for 
example the `minibuffer-completing-file-name' variable, which even uses a 
"neither true nor false" intermediate state in recursive minibuffers.

Anyway, I won't argue further, at this point it seems clear that you don't 
want the patch I proposed.

A remark on what you wrote yesterday:

>
> I'm not claiming that the changes I made yesterday and today are 
> supposed to produce the same effect as your proposed patch.  I'm just 
> making the display with overlay-string behave (as much as possible) like 
> display with normal buffer text, that's all.  Per bug#43519.  I'm not 
> saying that my changes implement the feature you are asking for here.
>

In fact these changes are, IMO, very regrettable, because they solve 95% 
of the problems that have been discussed in bug#24293, bug#39379, 
bug#43519 and bug#43572 (and perhaps others), and the remaining 5% will 
have to be solved another way (by text properties if that's what you agree 
on with Stefan).

This means that those who were trying to solve the problems in the 
above-mentioned bugs will be misled in thinking that they are now solved 
(if they don't immediately see these remaining 5%), and to not make the 
effort to use the (not yet implemented) correct solution.

I don't think you will do this, but please, please: revert these changes.

>> I'll wait until you and Stefan agree on the way to solve that problem 
>> in a better way to start working on this.  In his last mail he is 
>> apparently not sure anymore that using text properties to do this, as 
>> he suggested yesterday, is the best solution.
>
> Yes, I'm still unsure why Stefan said that, and am waiting for his 
> elaborations.
>

Okay, I'm waiting for the conclusion of that discussion.

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Eli Zaretskii]

> Date: Fri, 25 Sep 2020 08:34:50 +0000
> From: Gregory Heytings <ghe@sdf.org>
> cc: 43572@debbugs.gnu.org
> 
> > I remind you the fact that read_minibuf enters recursive-edit, during 
> > which any of the other callers of resize_mini_window can be called.
> 
> Now I think I understand what you mean.  Mini-windows are used for 
> minibuffers and for the echo area.  So for example, when 
> `start-display-at-beginning-of-minibuffer' is set while icomplete is 
> active in a frame, a call to `message' in another frame will display the 
> first part of the string instead of its last part if the string is too 
> large.

Yes.  And there are more callers of resize_mini_window than just
'message', so what you say above is just one such problematic
scenario.

> This seems like a really minor problem.

It is still a problem, and what looks minor to us might not be minor
in some valid use case out there.  IME, people write code and even
entire packages around some assumption about how Emacs works in
specific cases.  Changes that break those assumptions will not be
welcome, to say the least.

> >> If this case is important, the attached corrected patch also disables 
> >> setting `start-display-at-beginning-of-minibuffer' in recursive 
> >> minibuffers, that is, it limits the effect of that variable to 
> >> non-recursive minibuffers.
> >
> > That's a limitation I'd prefer not to impose.
> 
> I would also prefer not to impose that limitation, I added it because you 
> asked it (or at least that's what I understood).

I didn't ask for the limitation, I pointed out a problem with the
design you were proposing.  I'd prefer that the design and the
implementation of this feature did not impose such limitations.  Emacs
generally behaves the same on any level of minibuffer recursion, and
I'd like us not to violate that with this feature.

> > I'm not claiming that the changes I made yesterday and today are 
> > supposed to produce the same effect as your proposed patch.  I'm just 
> > making the display with overlay-string behave (as much as possible) like 
> > display with normal buffer text, that's all.  Per bug#43519.  I'm not 
> > saying that my changes implement the feature you are asking for here.
> 
> In fact these changes are, IMO, very regrettable, because they solve 95% 
> of the problems that have been discussed in bug#24293, bug#39379, 
> bug#43519 and bug#43572 (and perhaps others), and the remaining 5% will 
> have to be solved another way (by text properties if that's what you agree 
> on with Stefan).

They solve the issue pointed out by Stefan in bug#43519.  That they,
by sheer luck, also solve some of the other issues is just that --
sheer luck.  I don't claim and didn't intend to solve all the
problems, in particular the issue discussed in this bug report.  They
are related, but different issues.

> This means that those who were trying to solve the problems in the 
> above-mentioned bugs will be misled in thinking that they are now solved 
> (if they don't immediately see these remaining 5%), and to not make the 
> effort to use the (not yet implemented) correct solution.

I really don't see why that would be the case.  If someone is
motivated to solve whatever is left of those other problems, they will
solve them, or at least will point out which aspects of them remain
unresolved.

> I don't think you will do this, but please, please: revert these changes.

Reverting those changes would be a very strange thing to do.  Those
changes solve a specific problem, and they solve it cleanly.  That
other problems partially remain just means more changes are necessary.
In particular, the issues discussed here are a new feature which
didn't exist until now; reverting fixes for an existing problem
because they fail to introduce a new feature makes very little sense
to me.

Btw, there's nothing wrong in principle with installing a partial
solution for a problem (even though that's not what I meant to do with
the changes that resolve bug#43519).  We can and do decide sometimes
that it's a good idea to install a partial fix if the part fixed is
important enough, or if a comprehensive solution is too dangerous, or
for any other valid concern.  This is not uncommon in Emacs
development.

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> In fact these changes are, IMO, very regrettable, because they solve 
>> 95% of the problems that have been discussed in bug#24293, bug#39379, 
>> bug#43519 and bug#43572 (and perhaps others), and the remaining 5% will 
>> have to be solved another way (by text properties if that's what you 
>> agree on with Stefan).
>
> They solve the issue pointed out by Stefan in bug#43519.  That they, by 
> sheer luck, also solve some of the other issues is just that -- sheer 
> luck.  I don't claim and didn't intend to solve all the problems, in 
> particular the issue discussed in this bug report.  They are related, 
> but different issues.
>

There have been several misunderstandings in these discussions.  In 
bug#43519, Stefan pointed out an issue with a simple recipe to exhibit a 
more general problem.  This simple recipe, because it was simple, did not 
demonstrate all aspects of the problem.  In particular, it only 
demonstrated what he called "horizontal scrolling", when the problem in 
fact involves both horizontal _and vertical_ scrolling.

I'll remind a second time the discussion on emacs-devel that prompted 
Stefan to file bug#43519:

Ergus: [To implement icomplete-vertical] we need to add the exact amount 
of lines as accurate[ly] as possible.

Stefan: I *strongly* recommend you design the behavior under the 
assumption that it's OK if there are a few more lines in the (mini)buffer 
than are actually visible.

Me: If there are too many candidates the prompt disappears, leaving the 
cursor at the beginning of the minibuffer, which is counterintuitive.  A 
simple example: after (setq max-mini-window-height 1), with "M-x a" the 
"M-x" prompt and the "a" disappear.

Stefan: That can (and should) be fixed without having to reduce the number 
of candidates inserted in the (mini)buffer.

Ergus: It will be great if you give me an idea about how to do that.

Stefan: You need to figure out why the redisplay decides to hide the 
prompt rather than some other part of the (mini)buffer.

Stefan files bug#43519 (with the "M-x a" example).

As you see, the point is to keep the prompt visible, not just to avoid 
horizontal scrolling.  And Stefan refers to bug#24293 and bug#39379, where 
the same problem (the prompt becomes invisible) is explained and 
workarounds are discussed.

>> I don't think you will do this, but please, please: revert these 
>> changes.
>
> Reverting those changes would be a very strange thing to do.  Those 
> changes solve a specific problem, and they solve it cleanly.
>

They partially solve a specific problem.  This specific problem exists 
only in the context of inserting completion candidates at EOB with an 
overlay.  And the specific problem is not horizontal scrolling, the 
specific problem is the prompt that becomes invisible.  I clearly said 
(and explained) in bug#43519 that to solve that problem it is necessary to 
start display at BOB, and you preferred to implement a change that starts 
display at BOL.  I think these changes are misleading.

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Eli Zaretskii]

> Date: Fri, 25 Sep 2020 10:14:47 +0000
> From: Gregory Heytings <ghe@sdf.org>
> cc: 43572@debbugs.gnu.org
> 
> >> In fact these changes are, IMO, very regrettable, because they solve 
> >> 95% of the problems that have been discussed in bug#24293, bug#39379, 
> >> bug#43519 and bug#43572 (and perhaps others), and the remaining 5% will 
> >> have to be solved another way (by text properties if that's what you 
> >> agree on with Stefan).
> >
> > They solve the issue pointed out by Stefan in bug#43519.  That they, by 
> > sheer luck, also solve some of the other issues is just that -- sheer 
> > luck.  I don't claim and didn't intend to solve all the problems, in 
> > particular the issue discussed in this bug report.  They are related, 
> > but different issues.
> >
> 
> There have been several misunderstandings in these discussions.  In 
> bug#43519, Stefan pointed out an issue with a simple recipe to exhibit a 
> more general problem.  This simple recipe, because it was simple, did not 
> demonstrate all aspects of the problem.  In particular, it only 
> demonstrated what he called "horizontal scrolling", when the problem in 
> fact involves both horizontal _and vertical_ scrolling.

I'll leave it to Stefan to say what he meant, but here's my
understanding: the problem Stefan complained about in bug#43519 was
only what you call "horizontal scrolling", and he complained about it
because that didn't happen with text that came from a buffer (as
opposed to text from an overlay string).  This is the part that I
fixed.

What you call "vertical scrolling" happens with both buffer text and
overlay strings, and is the subject of this bug report.  It's a
separate issue.

> As you see, the point is to keep the prompt visible, not just to avoid 
> horizontal scrolling.

The prompt cannot always be kept visible as long as we display the
last portion of the text in the mini-window.  The code is explicitly
designed to enforce this, so it is not a bug, but the intended
behavior.  You suggest in this bug report to add a feature that allows
to change that policy.  But that is an issue separate from bug#43519,
and thus not directly related to the changes I installed to fix that
bug.

> And Stefan refers to bug#24293 and bug#39379, where the same problem
> (the prompt becomes invisible) is explained and workarounds are
> discussed.

I disagree that it's the same problem.  Part of what is described
there is the problem fixed in bug#43519, the other part is the
intended behavior.

> > Reverting those changes would be a very strange thing to do.  Those 
> > changes solve a specific problem, and they solve it cleanly.
> 
> They partially solve a specific problem.  This specific problem exists 
> only in the context of inserting completion candidates at EOB with an 
> overlay.  And the specific problem is not horizontal scrolling, the 
> specific problem is the prompt that becomes invisible.  I clearly said 
> (and explained) in bug#43519 that to solve that problem it is necessary to 
> start display at BOB, and you preferred to implement a change that starts 
> display at BOL.  I think these changes are misleading.

And I disagree, as explained above.  So let's agree to disagree,
because these back-and-forth messages, where we both say the same
things over and over and over again, are already too many to be
useful.

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> And Stefan refers to bug#24293 and bug#39379, where the same problem 
>> (the prompt becomes invisible) is explained and workarounds are 
>> discussed.
>
> I disagree that it's the same problem.  Part of what is described there 
> is the problem fixed in bug#43519, the other part is the intended 
> behavior.
>

Yet it is the same problem.  The fact that these bugs use simple examples 
(presumably because their authors did not yet investigate the more complex 
cases) does not mean that their bug reports mean something else than what 
they clearly say: "the prompt is not visible".

>
> And I disagree, as explained above.  So let's agree to disagree, because 
> these back-and-forth messages, where we both say the same things over 
> and over and over again, are already too many to be useful.
>

I agree to disagree.

Please tell me when you and Stefan agree on a better / the best way to 
solve the current problem.  I'll be silent till then.

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>>> Binding the variable inside the minibuffer-setup-hook will affect all 
>>> the subsequent calls to resize_mini_window, until the next call to 
>>> read-from-minibuffer resets it, which may not be what the Lisp program 
>>> wants, and could have unintended consequences.
>>
>> I can't think of such unintended consequences.  In the use case of 
>> displaying completion candidates, this (the fact that it affects all 
>> successive calls to resize_mini_window) is indeed what is wanted.
>
> Well, I _can_ think of such consequences.  As I said, resize_mini_window 
> is called in many situations that don't involve completion, so setting 
> that variable to affect all of them is a bad idea.  We need something 
> more fine-grained if we want to implement such a feature.
>

I believe I have dealt with these consequences (or at least with some of 
them) in the update patch I just sent.  Feel free to tell me if there are 
other cases in which resize_mini_window() is called with a wrong value for 
start_display_at_beginning_of_minibuffer.  It is now saved by 
read_minibuf(), and reset by read_minibuf() and read_minibuf_unwind().

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

A simple recipe to demonstrate the problem:

emacs -Q

(let (w bd)
   (setq w 60)
   (setq bd (concat (temporary-file-directory) (make-string w ?a) "/"))
   (dolist (d '("a" "b" "c" "d" "e")) (make-directory (concat bd d) t))
   (setq default-directory bd)
   (set-frame-height nil 20)
   (set-frame-width nil (+ (length bd) 10))
   (icomplete-mode)
   (setq icomplete-separator "\n")
   (call-interactively 'insert-file))

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Stefan Monnier]

> One case in which this behavior is not desirable is when completion
> candidates are displayed with an overlay at the end of the buffer.
> When this overlay is taller than max-mini-window-height, the prompt and the
> user input so far disappear.  A simple example: M-: (setq
> max-mini-window-height 1), M-x icomplete-mode, M-x a.

You should update your recipe, because Eli's patch takes care of this
case already.

I'm not completely sure which case(s) you're thinking of now that Eli's
patch handles the most common case we've seen so far.  But maybe the
problem shows up when we have a minibuffer content that spans 2 lines,
in which case the redisplay will choose to show the last line
(assuming point is in the second line) plus icomplete's overlay rather
whereas you presumably would want to see both lines from the minibuffer
(and hence one line less from icomplete's overlay)?

So a recipe could look something like:

    src/emacs -Q --eval '(setq max-mini-window-height 2)' -f icomplete-mode
    C-x C-f lisp/progmodes/../progmodes/../progmodes/../progmodes/../progmodes/../progmodes/../progmodes/../progmodes/../progmodes/../

where we see the whole of icomplete's overlay rather than seeing the
whole of the minibuffer's actual content.

In general both are perfectly valid choices and which one is best
depends on what is the intention behind the particular overlay and its
relation to the minibuffer's content, so indeed the redisplay would need
additional information in order to decide which behavior to choose.

> The attached patch makes it possible to (selectively) choose to display the
> _first_ lines of the minibuffer instead of its _last_ lines (which is and
> remains the default behavior).

Currently, the redisplay code focuses on making sure point is displayed.
In the resize_mini_widow code we try to accommodate some extra desires,
mostly in the form of giving more importance either to what's before point
or what's after it.

> The attached patch does not change the behavior of Emacs in any way,
> unless the feature it introduces is used.

I see the following potential problem with it: icomplete will likely
want to set it globally, but that means it will also affect uses of the
mini window where icomplete is not used.  Also, potential other users
may encounter similar difficulties.

I don't have a patch to suggest, but I think ideally, I'd want clients
like icomplete to tell the redisplay either something like "please
display as much as possible of *this* chunk of text" or maybe "feel free
not to display all of this overlay, it's not super important".

[ Note: "please display as much as possible of *this* chunk of text" is
  what I'd want to do in diff-mode when I move between chunks or in
  smerge-mode when I get to a new conflict, so maybe such a thing would
  make sense not just in the minibuffer.  ]


        Stefan

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Wed, 23 Sep 2020 14:33:13 -0400
> Cc: 43572@debbugs.gnu.org
> 
> I don't have a patch to suggest, but I think ideally, I'd want clients
> like icomplete to tell the redisplay either something like "please
> display as much as possible of *this* chunk of text" or maybe "feel free
> not to display all of this overlay, it's not super important".
> 
> [ Note: "please display as much as possible of *this* chunk of text" is
>   what I'd want to do in diff-mode when I move between chunks or in
>   smerge-mode when I get to a new conflict, so maybe such a thing would
>   make sense not just in the minibuffer.  ]

This could be a useful new feature, but we still need to decide what
should the display engine do when the chunk of text marked with this
new indication (some text property, probably?) cannot all of it be
displayed.  Should it then display only its first part, only its last
part, something else?

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Stefan Monnier]

> This could be a useful new feature, but we still need to decide what
> should the display engine do when the chunk of text marked with this
> new indication (some text property, probably?) cannot all of it be
> displayed.  Should it then display only its first part, only its last
> part, something else?

I think this can be controlled with the position of point (which
I guess should be presumed to be somewhere within the "important
chunk").  The idea would be to maximize the amount of chunk that's
displayed, but under the usual constraint that point is displayed.


        Stefan

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: ghe@sdf.org,  43572@debbugs.gnu.org
> Date: Wed, 23 Sep 2020 19:18:39 -0400
> 
> > This could be a useful new feature, but we still need to decide what
> > should the display engine do when the chunk of text marked with this
> > new indication (some text property, probably?) cannot all of it be
> > displayed.  Should it then display only its first part, only its last
> > part, something else?
> 
> I think this can be controlled with the position of point (which
> I guess should be presumed to be somewhere within the "important
> chunk").  The idea would be to maximize the amount of chunk that's
> displayed, but under the usual constraint that point is displayed.

Sure, point is important.  But it doesn't solve the problem I had in
mind.  Suppose you have text like this:

   xxxxxxxxxxxxx[xxxxxxxxxxxx|xxxxxxxxxxxxxxxxxx]xxxxxxxxxxx

where [...] denotes the portion of text indicated as "important
chunk", and | denotes the position of point.  Suppose further than the
available screen estate is insufficient to display all of the
"important chunk" -- which part would you want to see on display: the
part before point? after point? centered at point? something else?

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Stefan Monnier]

> where [...] denotes the portion of text indicated as "important
> chunk", and | denotes the position of point.  Suppose further than the
> available screen estate is insufficient to display all of the
> "important chunk" -- which part would you want to see on display: the
> part before point? after point? centered at point? something else?

I think either of those would be fine, so it should be decided by the
usual scrolling constraints (i.e. don't scroll if not needed, obey
scroll-conservatively, ...).

IOW, by default if scrolling was needed anyway and scroll-conservatively
is not set, I'd expect "centered at point".


        Stefan

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: ghe@sdf.org,  43572@debbugs.gnu.org
> Date: Thu, 24 Sep 2020 12:44:56 -0400
> 
> > where [...] denotes the portion of text indicated as "important
> > chunk", and | denotes the position of point.  Suppose further than the
> > available screen estate is insufficient to display all of the
> > "important chunk" -- which part would you want to see on display: the
> > part before point? after point? centered at point? something else?
> 
> I think either of those would be fine, so it should be decided by the
> usual scrolling constraints (i.e. don't scroll if not needed, obey
> scroll-conservatively, ...).
> 
> IOW, by default if scrolling was needed anyway and scroll-conservatively
> is not set, I'd expect "centered at point".

The code which implements automatic scrolling was not written with the
mini-window in mind.  In fact, we would like not to allow any
scrolling at all there.

So perhaps relying on scrolling could be fine in normal windows, it
will most probably do the wrong thing in mini-windows.  And even in
normal windows, it will probably work only by sheer luck, because the
design there cares about the context of point, and that is not
necessarily what you want with "important chunks".

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Stefan Monnier]

>> IOW, by default if scrolling was needed anyway and scroll-conservatively
>> is not set, I'd expect "centered at point".
> The code which implements automatic scrolling was not written with the
> mini-window in mind.  In fact, we would like not to allow any
> scrolling at all there.

When the window is big enough to show the whole content, I agree, but as
soon as the buffer is bigger then we need to handle scrolling, just
like elsewhere.

> So perhaps relying on scrolling could be fine in normal windows, it
> will most probably do the wrong thing in mini-windows.

That's not my experience so far.  I do find that the
`scroll-conservatively` is generally desired for the mini-window
(whereas I don't generally like it in normal windows), but other than
that I find (much to my surprise) that the generic scrolling code
behaves just as well as the ad-hoc scrolling code in resize_mini_window.

Thinking about it, maybe I shouldn't be surprised: the generic code is
used much more often and has been tuned quite heavily over the years to
provide good behavior is the vast majority of cases, so it's only
"normal" that it should behave nicely in this case as well.


        Stefan

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: ghe@sdf.org,  43572@debbugs.gnu.org
> Date: Sun, 27 Sep 2020 17:59:02 -0400
> 
> >> IOW, by default if scrolling was needed anyway and scroll-conservatively
> >> is not set, I'd expect "centered at point".
> > The code which implements automatic scrolling was not written with the
> > mini-window in mind.  In fact, we would like not to allow any
> > scrolling at all there.
> 
> When the window is big enough to show the whole content, I agree, but as
> soon as the buffer is bigger then we need to handle scrolling, just
> like elsewhere.
> 
> > So perhaps relying on scrolling could be fine in normal windows, it
> > will most probably do the wrong thing in mini-windows.
> 
> That's not my experience so far.  I do find that the
> `scroll-conservatively` is generally desired for the mini-window
> (whereas I don't generally like it in normal windows), but other than
> that I find (much to my surprise) that the generic scrolling code
> behaves just as well as the ad-hoc scrolling code in resize_mini_window.

Maybe it's 80% correct, but my worry is exactly those remaining 20%.
It is they that generate most of the bug reports in Emacs nowadays.
The fact that you find scroll-conservatively improve the results is
already a clear sign that the scrolling mechanism is not fit very well
for such small windows, especially when they are used for user
interaction.  And if you look at the window-display code, you will see
there several code fragments dealing with the cases of a text line
which is as tall or taller than the window, whose results are frankly
questionable, and whose only justification is that those cases "should
not happen".  What you suggest will make the probability of that
happening much higher, and in use cases where it will hurt.

And all this is even before we take the massive use of overlay strings
in some completion features.

So no, I cannot agree that your experience is evidence good enough for
allowing GP Emacs scrolling code to handle mini-window display.

> Thinking about it, maybe I shouldn't be surprised: the generic code is
> used much more often and has been tuned quite heavily over the years to
> provide good behavior is the vast majority of cases, so it's only
> "normal" that it should behave nicely in this case as well.

That is true, but it wasn't tuned to the use cases we meet every day
in the mini-windows.  We generally don't allow a window to become too
small in height, but that happens with mini-windows all the time.

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Stefan Monnier]

>> That's not my experience so far.  I do find that the
>> `scroll-conservatively` is generally desired for the mini-window
>> (whereas I don't generally like it in normal windows), but other than
>> that I find (much to my surprise) that the generic scrolling code
>> behaves just as well as the ad-hoc scrolling code in resize_mini_window.
> Maybe it's 80% correct, but my worry is exactly those remaining 20%.

Same here.  I only tried that code in very limited circumstances and
I know there are many more scenarios out there that I can't even begin
to imagine.

> So no, I cannot agree that your experience is evidence good enough for
> allowing GP Emacs scrolling code to handle mini-window display.

I definitely don't claim it's good enough to get rid of the ad-hoc
scrolling code, but I think it is worth pursuing.

How 'bout we add a config `minibuffer-scroll-generic`?


        Stefan

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: ghe@sdf.org,  43572@debbugs.gnu.org
> Date: Mon, 28 Sep 2020 09:30:46 -0400
> 
> How 'bout we add a config `minibuffer-scroll-generic`?

I don't mind, but who would use that, and why?

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Stefan Monnier]

>> How 'bout we add a config `minibuffer-scroll-generic`?
> I don't mind, but who would use that,

Those people I'd manage to con into setting it?

> and why?

To help collect info about it?


        Stefan

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Hi Stefan and Eli,

Thanks for your comments.

This feature request can be closed.  I found a solution that works without 
changing anything in Emacs core.

I still think that the changes introduced by Eli in xdisp.c should be 
reverted.  They are not a solution for the problem, and they change a 
long-standing behavior of a central part of Emacs, which might break or 
cause other problems with third-party packages that do unimaginable things 
with miniwindows.

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Stefan Monnier]

> I still think that the changes introduced by Eli in xdisp.c should be
> reverted.  They are not a solution for the problem, and they change
> a long-standing behavior of a central part of Emacs, which might break or
> cause other problems with third-party packages that do unimaginable things
> with miniwindows.

Like any change, Eli's change can introduce regressions, but so far
I haven't seen any clear regression, whereas we do know of one very
clear improvement.
So reverting would be ill-advised at this point.


        Stefan

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Eli Zaretskii]

> Date: Fri, 02 Oct 2020 15:40:42 +0000
> From: Gregory Heytings <ghe@sdf.org>
> 
> This feature request can be closed.  I found a solution that works without 
> changing anything in Emacs core.

As I wrote on emacs-devel, I don't like the solution you've found.
You are, of course, free to use it in your code, but I don't think we
should adopt it in core Emacs.

> I still think that the changes introduced by Eli in xdisp.c should be 
> reverted.

You've said this before, several times, and I explained why that would
be a bad idea, to say the least.  Reiterating that now sounds like
bullying, and why would you want to do something like that?

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> This feature request can be closed.  I found a solution that works 
>> without changing anything in Emacs core.
>
> As I wrote on emacs-devel, I don't like the solution you've found. You 
> are, of course, free to use it in your code, but I don't think we should 
> adopt it in core Emacs.
>

Your reaction is saddening.  I'm very sorry to read this, but you rejected 
all my attempts to solve that problem so far, so that's not surprising. 
I'd bet that there are many places in Emacs where similar "quick and 
dirty" hacks are used.  At least I did my best to solve that problem.

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Eli Zaretskii]

> Date: Fri, 02 Oct 2020 16:25:05 +0000
> From: Gregory Heytings <ghe@sdf.org>
> cc: monnier@iro.umontreal.ca, 43572@debbugs.gnu.org
> 
> Your reaction is saddening.  I'm very sorry to read this, but you rejected 
> all my attempts to solve that problem so far, so that's not surprising. 

I rejected what clearly looked like unsafe and unclean design.  Please
respect the opinions of those who have deeper knowledge of Emacs
internals and many years of experience hacking them.  Your insistence
on doing things in sub-optimal ways when much better ways are at hand
makes it hard to discuss these issues with you and arrive at sound
solutions that can be trusted not to produce bugs in the future.

> I'd bet that there are many places in Emacs where similar "quick and 
> dirty" hacks are used.

Actually, no, there aren't.  "Quick and dirty" is okay for rapid
prototyping and presenting a POC, but not for changes that get
admitted into core.

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> Your reaction is saddening.  I'm very sorry to read this, but you 
>> rejected all my attempts to solve that problem so far, so that's not 
>> surprising.
>
> I rejected what clearly looked like unsafe and unclean design.  Please 
> respect the opinions of those who have deeper knowledge of Emacs 
> internals and many years of experience hacking them.  Your insistence on 
> doing things in sub-optimal ways when much better ways are at hand makes 
> it hard to discuss these issues with you and arrive at sound solutions 
> that can be trusted not to produce bugs in the future.
>

I won't go in that direction, I will not start a public quarrel with you. 
I can only repeat that your reaction is saddening.

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Lars Ingebrigtsen]

Gregory Heytings <ghe@sdf.org> writes:

> This feature request can be closed.  I found a solution that works
> without changing anything in Emacs core.

(I'm going through old bug reports that unfortunately weren't resolved
at the time.)

Skimming this bug thread, it seems like there's room for improvement in
how Emacs displays these things, but it seems unlikely that there'll be
further progress here, so I'm closing this bug report.  The approach of
marking certain text for the display to pick up seems possible, but the
main use cases seem to be gone now (if I read correctly, which I may not
have).

If further work should be done here, I think opening a new bug report
and restating the aims would be the way forward.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> One case in which this behavior is not desirable is when completion 
>> candidates are displayed with an overlay at the end of the buffer. When 
>> this overlay is taller than max-mini-window-height, the prompt and the 
>> user input so far disappear.  A simple example: M-: (setq 
>> max-mini-window-height 1), M-x icomplete-mode, M-x a.
>
> You should update your recipe, because Eli's patch takes care of this 
> case already.
>

Indeed, but this recipe is what exists in Emacs 27.1 and what most people 
can try immediately.  The "problem" with Eli's patch is that it takes care 
of 95% of the cases, and that the remaining 5% are more difficult to see, 
yet are there.

>
> I'm not completely sure which case(s) you're thinking of now that Eli's 
> patch handles the most common case we've seen so far.  But maybe the 
> problem shows up when we have a minibuffer content that spans 2 lines, 
> in which case the redisplay will choose to show the last line (assuming 
> point is in the second line) plus icomplete's overlay rather whereas you 
> presumably would want to see both lines from the minibuffer (and hence 
> one line less from icomplete's overlay)?
>

Yes.  See the recipe in my previous mail.  If works with N lines, with N > 
1.

>
> So a recipe could look something like:
>
>    src/emacs -Q --eval '(setq max-mini-window-height 2)' -f icomplete-mode
>    C-x C-f lisp/progmodes/../progmodes/../progmodes/../progmodes/../progmodes/../progmodes/../progmodes/../progmodes/../progmodes/../
>
> where we see the whole of icomplete's overlay rather than seeing the 
> whole of the minibuffer's actual content.
>

The problem is that such a recipe would only work with a not-too-wide 
Emacs frame and a not-too-small default font.  For example, on my 
computer, it does not demonstrate the problem.

>
> In general both are perfectly valid choices and which one is best 
> depends on what is the intention behind the particular overlay and its 
> relation to the minibuffer's content, so indeed the redisplay would need 
> additional information in order to decide which behavior to choose.
>

I agree with you, which is why I think that ATM the best thing to do is 
just to make such a choice possible.

>
>> The attached patch makes it possible to (selectively) choose to display 
>> the _first_ lines of the minibuffer instead of its _last_ lines (which 
>> is and remains the default behavior).
>
> Currently, the redisplay code focuses on making sure point is displayed. 
> In the resize_mini_widow code we try to accommodate some extra desires, 
> mostly in the form of giving more importance either to what's before 
> point or what's after it.
>

I don't think this is precise enough, but OTOH it's very hard to precisely 
define what happens.  Currently resize_mini_window() sets "start" to 
max-mini-window-height lines before the end of the buffer (unless 
resize-mini-windows is nil, in which case "start" is set to BOB).  The 
redisplay code takes that value; if point is not between start and EOB it 
overrides it, otherwise it displays the buffer between start and EOB.

>
>> The attached patch does not change the behavior of Emacs in any way, 
>> unless the feature it introduces is used.
>
> I see the following potential problem with it: icomplete will likely 
> want to set it globally, but that means it will also affect uses of the 
> mini window where icomplete is not used.  Also, potential other users 
> may encounter similar difficulties.
>

No, if you look at the patch its value is reset to nil whenever the 
minibuffer is entered.  And the example I gave with icomplete is:

(add-hook 'icomplete-minibuffer-setup-hook (lambda () (setq start-display-at-beginning-of-minibuffer t)))

where icomplete-minibuffer-setup-hook is run during minibuffer setup if 
(and only if) icomplete is active.  So the behavior with M-: for example 
would not be affected.

>
> I don't have a patch to suggest, but I think ideally, I'd want clients 
> like icomplete to tell the redisplay either something like "please 
> display as much as possible of *this* chunk of text" or maybe "feel free 
> not to display all of this overlay, it's not super important".
>

Yes, that's the point.  The patch I proposed tells redisplay "please 
display as much as possible of the text between BOB and point", when the 
current behavior tells redisplay "please display as much as possible of 
the text between point and EOB".  Something more fine-grained would of 
course be possible, but I don't think it is a good idea to implement a 
feature when there is no clear need for it.

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Stefan Monnier]

>>    src/emacs -Q --eval '(setq max-mini-window-height 2)' -f icomplete-mode
>>    C-x C-f
>> lisp/progmodes/../progmodes/../progmodes/../progmodes/../progmodes/../progmodes/../progmodes/../progmodes/../progmodes/../
>>
>> where we see the whole of icomplete's overlay rather than seeing the whole
>> of the minibuffer's actual content.
>>
>
> The problem is that such a recipe would only work with a not-too-wide Emacs
> frame and a not-too-small default font.  For example, on my computer, it
> does not demonstrate the problem.

Really?  Despite the `-Q`?  Is it using a proportional font?
How many columns wide is the frame?
Oh, wait, is it because you're running that Emacs session in a tty
rather than in a GUI?

>>> The attached patch does not change the behavior of Emacs in any way,
>>> unless the feature it introduces is used.
>> I see the following potential problem with it: icomplete will likely want
>> to set it globally, but that means it will also affect uses of the mini
>> window where icomplete is not used.  Also, potential other users may
>> encounter similar difficulties.
> No, if you look at the patch its value is reset to nil whenever the
> minibuffer is entered.

That still leaves the non-minibuffer uses of the mini window (i.e. the
echo area).

And of course, that will also result in the use of an incorrect value
when you exit a *nested* minibuffer (unless both minibuffers were using
the same value).

>> I don't have a patch to suggest, but I think ideally, I'd want clients
>> like icomplete to tell the redisplay either something like "please display
>> as much as possible of *this* chunk of text" or maybe "feel free not to
>> display all of this overlay, it's not super important".
>
> Yes, that's the point.  The patch I proposed tells redisplay "please display
> as much as possible of the text between BOB and point", when the current
> behavior tells redisplay "please display as much as possible of the text
> between point and EOB".  Something more fine-grained would of course be
> possible, but I don't think it is a good idea to implement a feature when
> there is no clear need for it.

I think what I was getting at is that this "request" should come from
the minibuffer's text rather than from a variable.


        Stefan

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[-- Attachment #1: Type: text/plain, Size: 1998 bytes --]


>
> Really?  Despite the `-Q`?  Is it using a proportional font?
> Oh, wait, is it because you're running that Emacs session in a tty rather than in a GUI?
>

Yes, yes, no, no ;-)

>
> How many columns wide is the frame?
>

167, Sir! ;-)

>> No, if you look at the patch its value is reset to nil whenever the 
>> minibuffer is entered.
>
> That still leaves the non-minibuffer uses of the mini window (i.e. the 
> echo area).
>
> And of course, that will also result in the use of an incorrect value 
> when you exit a *nested* minibuffer (unless both minibuffers were using 
> the same value).
>

Thank you very much for the pointers, indeed I completely forgot about 
these cases.  See the attached corrected patch, in which they are dealt 
with.  I believe start_display_at_beginning_of_minibuffer how has the 
value it should have whenever resize_mini_window() is called.

>>> I don't have a patch to suggest, but I think ideally, I'd want clients 
>>> like icomplete to tell the redisplay either something like "please 
>>> display as much as possible of *this* chunk of text" or maybe "feel 
>>> free not to display all of this overlay, it's not super important".
>>
>> Yes, that's the point.  The patch I proposed tells redisplay "please 
>> display as much as possible of the text between BOB and point", when 
>> the current behavior tells redisplay "please display as much as 
>> possible of the text between point and EOB".  Something more 
>> fine-grained would of course be possible, but I don't think it is a 
>> good idea to implement a feature when there is no clear need for it.
>
> I think what I was getting at is that this "request" should come from 
> the minibuffer's text rather than from a variable.
>

If this were possible, it would be even better indeed.  An "importance" 
text property, which would inform redisplay (?) of the relative importance 
of the parts of the buffer.  But implementing this is several orders of 
magnitude harder than implementing my proposal.

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-diff; name=start-display-at-beginning-of-minibuffer.patch, Size: 2954 bytes --]

diff --git a/src/minibuf.c b/src/minibuf.c
index f957b2ae17..691a8c4f9b 100644
--- a/src/minibuf.c
+++ b/src/minibuf.c
@@ -472,6 +472,8 @@ read_minibuf (Lisp_Object map, Lisp_Object initial, Lisp_Object prompt,
     = Fcons (Voverriding_local_map,
 	     Fcons (minibuf_window,
 		    minibuf_save_list));
+  minibuf_save_list
+    = Fcons (Vstart_display_at_beginning_of_minibuffer, minibuf_save_list);
   minibuf_save_list
     = Fcons (minibuf_prompt,
 	     Fcons (make_fixnum (minibuf_prompt_width),
@@ -494,6 +496,7 @@ read_minibuf (Lisp_Object map, Lisp_Object initial, Lisp_Object prompt,
 
   /* Now that we can restore all those variables, start changing them.  */
 
+  Vstart_display_at_beginning_of_minibuffer = Qnil;
   minibuf_prompt_width = 0;
   minibuf_prompt = Fcopy_sequence (prompt);
   Vminibuffer_history_position = histpos;
@@ -802,6 +805,8 @@ read_minibuf_unwind (void)
   minibuf_save_list = Fcdr (minibuf_save_list);
   Vminibuffer_history_variable = Fcar (minibuf_save_list);
   minibuf_save_list = Fcdr (minibuf_save_list);
+  Vstart_display_at_beginning_of_minibuffer = Fcar (minibuf_save_list);
+  minibuf_save_list = Fcdr (minibuf_save_list);
   Voverriding_local_map = Fcar (minibuf_save_list);
   minibuf_save_list = Fcdr (minibuf_save_list);
 #if 0
@@ -2097,6 +2102,15 @@ syms_of_minibuf (void)
 uses to hide passwords.  */);
   Vread_hide_char = Qnil;
 
+  DEFVAR_LISP ("start-display-at-beginning-of-minibuffer", Vstart_display_at_beginning_of_minibuffer,
+	       doc: /* Whether to preferably display the beginning of the minibuffer.
+When the mini-window is not large enough to display the complete minibuffer,
+the default behavior is to display the last part of the minibuffer and to
+hide its first part.  This variable reverses that behavior.  This variable
+is reset before and after each read from the minibuffer, and must be set in
+`minibuffer-setup-hook'.  */);
+  Vstart_display_at_beginning_of_minibuffer = Qnil;
+
   defsubr (&Sactive_minibuffer_window);
   defsubr (&Sset_minibuffer_window);
   defsubr (&Sread_from_minibuffer);
diff --git a/src/xdisp.c b/src/xdisp.c
index 49225c56fe..1b1a34b97f 100644
--- a/src/xdisp.c
+++ b/src/xdisp.c
@@ -11804,7 +11804,7 @@ resize_mini_window (struct window *w, bool exact_p)
 	}
 
       /* Compute a suitable window start.  */
-      if (height > max_height)
+      if (height > max_height && !EQ (Vstart_display_at_beginning_of_minibuffer, Qt))
 	{
 	  height = (max_height / unit) * unit;
 	  init_iterator (&it, w, ZV, ZV_BYTE, NULL, DEFAULT_FACE_ID);
@@ -11812,7 +11812,10 @@ resize_mini_window (struct window *w, bool exact_p)
 	  start = it.current.pos;
 	}
       else
-	SET_TEXT_POS (start, BEGV, BEGV_BYTE);
+	{
+	  if (height > max_height) height = (max_height / unit) * unit;
+	  SET_TEXT_POS (start, BEGV, BEGV_BYTE);
+	}
 
       SET_MARKER_FROM_TEXT_POS (w->start, start);
 

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Stefan Monnier]

>> Really?  Despite the `-Q`?  Is it using a proportional font?
>> Oh, wait, is it because you're running that Emacs session in a tty rather than in a GUI?
> Yes, yes, no, no ;-)
>> How many columns wide is the frame?
> 167, Sir! ;-)

How? why?


        Stefan

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>>> Really?  Despite the `-Q`?  Is it using a proportional font?
>>> Oh, wait, is it because you're running that Emacs session in a tty rather than in a GUI?
>>
>> Yes, yes, no, no ;-)
>>
>>> How many columns wide is the frame?
>>
>> 167, Sir! ;-)
>
> How? why?
>

Simply because I asked my window manager to always maximize Emacs frames.

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Stefan Monnier]

> Simply because I asked my window manager to always maximize Emacs frames.

Aha!

Yet another loophole!
Thanks,


        Stefan

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> Simply because I asked my window manager to always maximize Emacs 
>> frames.
>
> Aha!
>
> Yet another loophole!
>

Hmm...  Given the popularity (among a certain kind of users) of tiling 
window managers and the popularity (among a much larger kind of users) of 
using fullscreen apps, I don't think Emacs can expect to fully control the 
size of its frame.  But indeed I see what you mean, ideally emacs -Q 
should give a frame of the same size everywhere.  So it could perhaps make 
sense to try to do something like:

(set-frame-width nil 80)
(set-frame-height nil 40)

(which works for me) at the end of the initialization process with "-q" or 
"-Q".

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Stefan Monnier]

>>> Simply because I asked my window manager to always maximize Emacs frames.
>> Aha!
>> Yet another loophole!
> Hmm...  Given the popularity (among a certain kind of users) of tiling
> window managers and the popularity (among a much larger kind of users) of
> using fullscreen apps, I don't think Emacs can expect to fully control the
> size of its frame.  But indeed I see what you mean, ideally emacs -Q should
> give a frame of the same size everywhere.  So it could perhaps make sense to
> try to do something like:
>
> (set-frame-width nil 80)
> (set-frame-height nil 40)
>
> (which works for me) at the end of the initialization process with "-q" or
>  "-Q".

I definitely wouldn't want that for `-q`.

It might make sense in this case for `-Q`, but I'm not sure it's a good
tradeoff since we also want `emacs -Q` to mean something like "run with
no special configuration".

So maybe a better option is for the recipe to specify the frame's width.


        Stefan

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[-- Attachment #1: Type: text/plain, Size: 889 bytes --]


To illustrate the need for this feature, I attach four screenshots taken 
under the exact same conditions.

- "vanilla.png" is what a user would see without changing any default 
(after pressing TAB to display the completion candidates): the prompt, 
their input, and completion candidates in a *Completions* buffer above the 
minibuffer

- "master-icomplete-vertical-after-tab.png" is what a user would see with 
icomplete-mode activated and (setq icomplete-separator "\n") after having 
completed the last directory name with a TAB: the prompt and their input 
is completely hidden

- "master-icomplete-vertical-after-tab-tab.png" is what a user would see 
after pressing TAB a second time; note the "ng/" before point, which was 
hidden after pressing TAB for the first time and is now visible again

- "patched-icomplete-vertical.png" is what a user would see with the 
proposed patch

[-- Attachment #2: Type: image/png, Size: 50216 bytes --]

[-- Attachment #3: Type: image/png, Size: 28832 bytes --]

[-- Attachment #4: Type: image/png, Size: 62417 bytes --]

[-- Attachment #5: Type: image/png, Size: 35481 bytes --]

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Eli Zaretskii]

> Date: Thu, 24 Sep 2020 08:06:37 +0000
> From: Gregory Heytings <ghe@sdf.org>
> cc: 43572@debbugs.gnu.org
> 
> - "master-icomplete-vertical-after-tab.png" is what a user would see with 
> icomplete-mode activated and (setq icomplete-separator "\n") after having 
> completed the last directory name with a TAB: the prompt and their input 
> is completely hidden

Thanks, this subtle issue should now be fixed on master.

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>> - "master-icomplete-vertical-after-tab.png" is what a user would see 
>> with icomplete-mode activated and (setq icomplete-separator "\n") after 
>> having completed the last directory name with a TAB: the prompt and 
>> their input is completely hidden
>
> Thanks, this subtle issue should now be fixed on master.
>

It is not, the only difference is that instead of "the prompt and their 
input is completely hidden" one should now write "the prompt and the 
largest part of their input is completely hidden".

Again, could you please provide a recipe which would demonstrate a problem 
with the technique I propose?

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Eli Zaretskii]

> Date: Thu, 24 Sep 2020 14:52:38 +0000
> From: Gregory Heytings <ghe@sdf.org>
> cc: monnier@iro.umontreal.ca, 43572@debbugs.gnu.org
> 
> >> - "master-icomplete-vertical-after-tab.png" is what a user would see 
> >> with icomplete-mode activated and (setq icomplete-separator "\n") after 
> >> having completed the last directory name with a TAB: the prompt and 
> >> their input is completely hidden
> >
> > Thanks, this subtle issue should now be fixed on master.
> 
> It is not, the only difference is that instead of "the prompt and their 
> input is completely hidden" one should now write "the prompt and the 
> largest part of their input is completely hidden".

I mean the issue with the prompt being hidden after one TAB, but
appear after another TAB.

All the rest is just the consequence of the current policy to show the
last part of the stuff in the mini-window.  (Note that with your
changes, one of the candidates isn't shown in the mini-window.)

There might be a misunderstanding here: I'm not claiming that the
changes I made yesterday and today are supposed to produce the same
effect as your proposed patch.  I'm just making the display with
overlay-string behave (as much as possible) like display with normal
buffer text, that's all.  Per bug#43519.  I'm not saying that my
changes implement the feature you are asking for here.

> Again, could you please provide a recipe which would demonstrate a problem 
> with the technique I propose?

I already explained why your design is problematic.  And so did
Stefan.  So I don't see why a recipe would be needed.

Implementation of such a feature should use text and/or overlay
properties to communicate the intent to the display code.  That would
be much cleaner and safer.

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Eli Zaretskii]

> Cc: 43572@debbugs.gnu.org
> Date: Wed, 23 Sep 2020 22:47:05 +0000
> From: Gregory Heytings via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> > I think what I was getting at is that this "request" should come from 
> > the minibuffer's text rather than from a variable.
> 
> If this were possible, it would be even better indeed.  An "importance" 
> text property, which would inform redisplay (?) of the relative importance 
> of the parts of the buffer.  But implementing this is several orders of 
> magnitude harder than implementing my proposal.

Actually testing for a certain text property in resize_mini_window
should be quite simple.  We do similar things elsewhere in the display
code.  So I see no reason to avoid that.  It would definitely resolve
any issues with inadvertently affecting unrelated users of the
mini-window.

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Eli Zaretskii]

> Cc: 43572@debbugs.gnu.org
> Date: Wed, 23 Sep 2020 19:46:18 +0000
> From: Gregory Heytings via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> >> The attached patch does not change the behavior of Emacs in any way, 
> >> unless the feature it introduces is used.
> >
> > I see the following potential problem with it: icomplete will likely 
> > want to set it globally, but that means it will also affect uses of the 
> > mini window where icomplete is not used.  Also, potential other users 
> > may encounter similar difficulties.
> 
> No, if you look at the patch its value is reset to nil whenever the 
> minibuffer is entered.  And the example I gave with icomplete is:
> 
> (add-hook 'icomplete-minibuffer-setup-hook (lambda () (setq start-display-at-beginning-of-minibuffer t)))
> 
> where icomplete-minibuffer-setup-hook is run during minibuffer setup if 
> (and only if) icomplete is active.  So the behavior with M-: for example 
> would not be affected.

There are more callers of resize_mini_widow than just those.  It is
not safe to assume that this can be handled only inside read_minibuf.

So I agree with Stefan that the text inserted into the minibuffer
should itself indicate to the display engine that it wants to be
displayed starting at BOB.  That way we don't have to worry about
inadvertently affecting other users of the mini-window.

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Gregory Heytings via Bug reports for GNU Emacs, the Swiss army knife of text editors]

>
> There are more callers of resize_mini_widow than just those.  It is not 
> safe to assume that this can be handled only inside read_minibuf.
>

Please have a look at my corrected patch.  It handles this inside 
read_minibuf() *and* read_minibuf_unwind(), which AFAICS should make the 
change to start_display_at_beginning_of_minibuffer last only for the time 
it should last.  If that's not the case, could you please provide a 
recipe?  I'm willing to improve the patch a second time if it's necessary.

>
> So I agree with Stefan that the text inserted into the minibuffer should 
> itself indicate to the display engine that it wants to be displayed 
> starting at BOB.  That way we don't have to worry about inadvertently 
> affecting other users of the mini-window.
>

That's a possibility indeed, and I agree that it would be even better, but 
as I said this is several orders of magnitude harder to implement, and 
unlikely to happen in a near future.

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Stefan Monnier]

> So I agree with Stefan that the text inserted into the minibuffer
> should itself indicate to the display engine that it wants to be
> displayed starting at BOB.  That way we don't have to worry about
> inadvertently affecting other users of the mini-window.

It might be difficult/inconvenient to have the info directly in the
text, tho.  Having it in a variable is not great (one of the
problems with it is that it depends on things like point so it's really
more "per-window" whereas variables are "per-buffer"), tho for the case
of minibuffers at least those buffers are normally never shown in more
than one window.

How 'bout using a window-parameter whose value should be an overlay
indicating the "area of focus", and then only obey this parameter if:
- the overlay is in the buffer that's being displayed.
- and window-point is lies within this overlay.

One more thing: there's a good argument to make that icomplete-vertical
should list the completions *above* the minibuffer's prompt rather than
below (so as not to affect the positions of the minibuffer's prompt so
much).  But in that case, the part of the overlay's after/before string
(when too long) which should be truncated (when the mini-window is too
small) is the beginning, whereas IIUC the current redisplay is unable to
display "the end" of an overlay's after/before string unless it also
shows its beginning.


        Stefan

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Gregory Heytings <ghe@sdf.org>,  43572@debbugs.gnu.org
> Date: Thu, 24 Sep 2020 12:40:59 -0400
> 
> > So I agree with Stefan that the text inserted into the minibuffer
> > should itself indicate to the display engine that it wants to be
> > displayed starting at BOB.  That way we don't have to worry about
> > inadvertently affecting other users of the mini-window.
> 
> It might be difficult/inconvenient to have the info directly in the
> text, tho.

I don't think I see why it would be difficult/inconvenient.  Can you
explain?

> How 'bout using a window-parameter whose value should be an overlay
> indicating the "area of focus", and then only obey this parameter if:
> - the overlay is in the buffer that's being displayed.
> - and window-point is lies within this overlay.

This sounds like the same idea of a text property, only with an
overlay and a more complicated test for applicability.  When I said
"text property", I meant both text and overlay property, so if you
think your proposal above is less difficult/inconvenient, then using
just an overlay would be even simpler, no?  Or what am I missing?

> One more thing: there's a good argument to make that icomplete-vertical
> should list the completions *above* the minibuffer's prompt rather than
> below (so as not to affect the positions of the minibuffer's prompt so
> much).  But in that case, the part of the overlay's after/before string
> (when too long) which should be truncated (when the mini-window is too
> small) is the beginning, whereas IIUC the current redisplay is unable to
> display "the end" of an overlay's after/before string unless it also
> shows its beginning.

Yes, we cannot start a window's display in the middle of an overlay
string.

In general, display and overlay strings were not intended for 75%
(maybe more) of the uses they get nowadays, and it shows.

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Eli Zaretskii]

Ping!

> Date: Thu, 24 Sep 2020 19:54:36 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: ghe@sdf.org, 43572@debbugs.gnu.org
> 
> > From: Stefan Monnier <monnier@iro.umontreal.ca>
> > Cc: Gregory Heytings <ghe@sdf.org>,  43572@debbugs.gnu.org
> > Date: Thu, 24 Sep 2020 12:40:59 -0400
> > 
> > > So I agree with Stefan that the text inserted into the minibuffer
> > > should itself indicate to the display engine that it wants to be
> > > displayed starting at BOB.  That way we don't have to worry about
> > > inadvertently affecting other users of the mini-window.
> > 
> > It might be difficult/inconvenient to have the info directly in the
> > text, tho.
> 
> I don't think I see why it would be difficult/inconvenient.  Can you
> explain?
> 
> > How 'bout using a window-parameter whose value should be an overlay
> > indicating the "area of focus", and then only obey this parameter if:
> > - the overlay is in the buffer that's being displayed.
> > - and window-point is lies within this overlay.
> 
> This sounds like the same idea of a text property, only with an
> overlay and a more complicated test for applicability.  When I said
> "text property", I meant both text and overlay property, so if you
> think your proposal above is less difficult/inconvenient, then using
> just an overlay would be even simpler, no?  Or what am I missing?
> 
> > One more thing: there's a good argument to make that icomplete-vertical
> > should list the completions *above* the minibuffer's prompt rather than
> > below (so as not to affect the positions of the minibuffer's prompt so
> > much).  But in that case, the part of the overlay's after/before string
> > (when too long) which should be truncated (when the mini-window is too
> > small) is the beginning, whereas IIUC the current redisplay is unable to
> > display "the end" of an overlay's after/before string unless it also
> > shows its beginning.
> 
> Yes, we cannot start a window's display in the middle of an overlay
> string.
> 
> In general, display and overlay strings were not intended for 75%
> (maybe more) of the uses they get nowadays, and it shows.

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Stefan Monnier]

>> This sounds like the same idea of a text property, only with an
>> overlay and a more complicated test for applicability.  When I said
>> "text property", I meant both text and overlay property, so if you
>> think your proposal above is less difficult/inconvenient, then using
>> just an overlay would be even simpler, no?  Or what am I missing?

My motivation was to solve the issue of finding the relevant overlay.
But yes, we could also just try and find "the overlay that covers point
and which has a non-nil `scroll-focus` property".  We'll just have to be
careful to properly handle the case where point is exactly at
`overlay-start` or `overlay-end`.


        Stefan

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: ghe@sdf.org,  43572@debbugs.gnu.org
> Date: Sun, 27 Sep 2020 17:52:21 -0400
> 
> >> This sounds like the same idea of a text property, only with an
> >> overlay and a more complicated test for applicability.  When I said
> >> "text property", I meant both text and overlay property, so if you
> >> think your proposal above is less difficult/inconvenient, then using
> >> just an overlay would be even simpler, no?  Or what am I missing?
> 
> My motivation was to solve the issue of finding the relevant overlay.
> But yes, we could also just try and find "the overlay that covers point
> and which has a non-nil `scroll-focus` property".  We'll just have to be
> careful to properly handle the case where point is exactly at
> `overlay-start` or `overlay-end`.

I actually thought about putting the text property or overlay on the
first character of the text inserted into the minibuffer, something
that can be done in a minibuffer-setup-hook.  This way, it will be
easy to find.  Of course, having that on any other part of the text
will not make the job much harder, either.

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Stefan Monnier]

> I actually thought about putting the text property or overlay on the
> first character of the text inserted into the minibuffer,

I don't understand what data you intend to put this way.
The overlay I proposed carries fundamentally 2 pieces of info: the
beginning position and the end position of the "scroll focus".
So if it needs to cover the first char, you can't use `overlay-start` to
carry the first info.


        Stefan

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: ghe@sdf.org,  43572@debbugs.gnu.org
> Date: Mon, 28 Sep 2020 09:24:34 -0400
> 
> > I actually thought about putting the text property or overlay on the
> > first character of the text inserted into the minibuffer,
> 
> I don't understand what data you intend to put this way.

Just "start at BOB" indication.

> The overlay I proposed carries fundamentally 2 pieces of info: the
> beginning position and the end position of the "scroll focus".

We could still put this on the first character, and have the value of
the property be a cons cell with these two positions, no?

IOW, the information doesn't have to be on the characters which it
describes.

bug#43572: Feature request: make it possible to choose whether the first lines of the minibuffer should be displayed instead of the last ones

[Stefan Monnier]

>> The overlay I proposed carries fundamentally 2 pieces of info: the
>> beginning position and the end position of the "scroll focus".
> We could still put this on the first character,

Indeed, that's an option, but it doesn't seem particularly handy.
Not necessarily worse than a window-property or a buffer-local value,
admittedly.

> and have the value of the property be a cons cell with these two
> positions, no?

An overlay is a handy abstraction for "pair of two positions in the same
buffer", with extra niceties such as the `evaporate` property.


        Stefan