A solution to display completion candidates after point in a minibuffer

A solution to display completion candidates after point in a minibuffer

[Gregory Heytings via Emacs development discussions.]

Hi,

Displaying completion candidates after point in a minibuffer has raised a 
similar problem over the last years (see for instance bug#24293, 
bug#39379, bug#43519 and bug#43572): when there are too many completion 
candidates, the minibuffer prompt is hidden.  The root of this problem is 
that resize_mini_window() (in xdisp.c) computes the start of the visible 
part of the minibuffer so as to make the end of the minibuffer contents 
appear.

So far the solution to this problem has been to manually compute the size 
of the completion candidates so that they do not use (together with the 
prompt and the user input) more than max-mini-window-height lines. 
Another solution, which as far as I know has not been used, would be to 
set resize-mini-windows to nil, and to resize the miniwindow manually with 
enlarge-window, but this solution also involves nontrivial computations, 
and fiddles with a user setting.

The good news is that it is in fact possible to convince Emacs to do the 
opposite of what resize_mini_window() does:

(defvar-local start-display-at-beginning-of-minibuffer nil)
(defun start-display-at-beginning-of-minibuffer (&rest args)
   (when (and start-display-at-beginning-of-minibuffer (minibufferp))
     (set-window-start-at-begin 1 (point))))
(defun set-window-start-at-begin (beg end)
   (set-window-start nil beg)
   (unless (pos-visible-in-window-p end nil t) (set-window-start-at-begin (+ beg (/ (- end beg) 2)) end)))
(setq window-scroll-functions (cons 'start-display-at-beginning-of-minibuffer window-scroll-functions))
(add-hook 'post-command-hook 'start-display-at-beginning-of-minibuffer)

This works with at least Emacs 24, 25, 26, 27 and 28.

This means that displaying completion candidates after point in a 
minibuffer becomes a trivial task: it suffices to insert the completion 
candidates in the minibuffer, without worrying at all about their size (or 
about the size of the prompt and user input), and Emacs will display as 
many of these candidates as possible, given the user preferences 
(max-mini-window-height, resize-mini-windows, ...).

As a proof of concept, displaying completion candidates vertically with 
icomplete is as easy as:

(setq icomplete-separator "\n")
(add-hook 'icomplete-minibuffer-setup-hook (lambda () (setq start-display-at-beginning-of-minibuffer t)))
(defun icomplete-vertical-reformat-completions (completions)
   (save-match-data
     (if (string-match "^\\((.*)\\|\\[.+\\]\\)?{\\(\\(?:.\\|\n\\)+\\)}" completions)
         (format "%s \n%s" (or (match-string 1 completions) "") (match-string 2 completions))
       completions)))
(advice-add 'icomplete-completions :filter-return #'icomplete-vertical-reformat-completions)

A few comments, for the curious:

1. Obviously, when the miniwindow is forced to be small (for example with 
(setq max-mini-window-height 1)), the prompt will be hidden when the 
prompt and user input are too large.

2. The only drawback of the above solution is that is is not possible to 
display an ellipsis ("...") at the end of the completion candidates list, 
to indicate that some completion candidates are not displayed.  It seems 
to me that this is a minor limitation.

3. If the face used in the minibuffer has a height that is not a multiple 
of the height of the default face, the last completion candidate will be 
only partially visible.

4. set-window-start-at-begin is very fast, on my (not very recent) 
computer it takes about a half millisecond.  In the normal case, that is, 
when the miniwindow is not too small, set-window-start-at-begin merely 
does (set-window-start nil 1).

5. set-better-window-start could in theory enter an infinite loop (that 
is, raise a "Lisp nesting exceeds 'max-lisp-eval-depth'" error) if setting 
window-start near point failed.  This cannot happen in practice.

6. Instead of using (+ beg (/ (- end beg) 2)) in set-better-window-start, 
one could be tempted to use (window-body-width), but this does not work 
when the face used in the minibuffer is not the default face.  One could 
also be tempted to use (/ (window-text-width nil t) (car 
(window-text-pixel-size nil 1 2))), but this does not work with a variable 
width face.

7. (add-hook 'post-command-hook 'start-display-at-beginning-of-minibuffer) 
is necessary only with variable width faces, but it is does not harm to 
use it with fixed width faces.

8. I believe there is only one thing missing in the "vertical icomplete" 
implementation above, namely removing the first line when it is empty. 
This can be done as follows:

(defun icomplete-vertical-reformat-completions (completions)
   (save-match-data
     (if (string-match "^\\((.*)\\|\\[.+\\]\\)?{\\(\\(?:.\\|\n\\)+\\)}" completions)
 	(let* ((c (match-string 2 completions))
 	       (nl (= (aref c 0) 10))
 	       (cc (if nl (substring c 1) c)))
           (format "%s \n%s" (or (match-string 1 completions) "") cc))
       completions)))

Gregory

Re: A solution to display completion candidates after point in a minibuffer

[Eli Zaretskii]

> Date: Fri, 02 Oct 2020 15:36:37 +0000
> From: Gregory Heytings via "Emacs development discussions." <emacs-devel@gnu.org>
> 
> (defvar-local start-display-at-beginning-of-minibuffer nil)
> (defun start-display-at-beginning-of-minibuffer (&rest args)
>    (when (and start-display-at-beginning-of-minibuffer (minibufferp))
>      (set-window-start-at-begin 1 (point))))
> (defun set-window-start-at-begin (beg end)
>    (set-window-start nil beg)
>    (unless (pos-visible-in-window-p end nil t) (set-window-start-at-begin (+ beg (/ (- end beg) 2)) end)))
> (setq window-scroll-functions (cons 'start-display-at-beginning-of-minibuffer window-scroll-functions))
> (add-hook 'post-command-hook 'start-display-at-beginning-of-minibuffer)
> 
> This works with at least Emacs 24, 25, 26, 27 and 28.

Thanks, but I don't like such solutions, because they attempt to work
around the display engine in order to do something, instead of asking
the display engine itself to do that for us.

In bug#43572, a solution that would allow us to ask the display engine
to do this is being discussed.  I'd favor one of the solutions
proposed there lately, namely, to indicate in the text to be displayed
that this text wants to be displayed with the window-start set at BOB.

Re: A solution to display completion candidates after point in a minibuffer

[Gregory Heytings via Emacs development discussions.]

>
> Thanks, but I don't like such solutions, because they attempt to work 
> around the display engine in order to do something, instead of asking 
> the display engine itself to do that for us.
>

You may not like it, but it works.  In many cases programming means 
finding an appropriate workaround, and this workaround is better than 
those have been used so far.

>
> In bug#43572, a solution that would allow us to ask the display engine 
> to do this is being discussed.  I'd favor one of the solutions proposed 
> there lately, namely, to indicate in the text to be displayed that this 
> text wants to be displayed with the window-start set at BOB.
>

This solution, on which there is no agreement yet, still has to be 
implemented, and it will only work, in the best case, with Emacs 27.2 and 
later.  The solution here works with all Emacs versions.

Re: A solution to display completion candidates after point in a minibuffer

[Eli Zaretskii]

> Date: Fri, 02 Oct 2020 16:14:07 +0000
> From: Gregory Heytings <ghe@sdf.org>
> cc: emacs-devel@gnu.org
> 
> > In bug#43572, a solution that would allow us to ask the display engine 
> > to do this is being discussed.  I'd favor one of the solutions proposed 
> > there lately, namely, to indicate in the text to be displayed that this 
> > text wants to be displayed with the window-start set at BOB.
> 
> This solution, on which there is no agreement yet, still has to be 
> implemented, and it will only work, in the best case, with Emacs 27.2 and 
> later.  The solution here works with all Emacs versions.

I'm not against using this as a workaround, if someone needs it, until
the better solution arrives.  I was just saying that using that code
_instead_ of a solution in core Emacs would be wrong, IMO.

Re: A solution to display completion candidates after point in a minibuffer

[Stefan Monnier]

I think you should better define the problem you're trying to solve.

AFAIK the main issue ("when there are too many completion candidates,
the minibuffer prompt is hidden") has been solved by Eli, so without
some concrete cases where the result is still unsatisfactory it's
difficult to know what "better" means.  The code can't guess what should
happen in all cases currently because the information for that is simply
not available, so it remains a balancing act based on heuristics.

[ I sightly tweaked your code ]
> (defvar-local start-display-at-beginning-of-minibuffer nil)
> (defun start-display-at-beginning-of-minibuffer (&rest args)
>   (when (and start-display-at-beginning-of-minibuffer (minibufferp))
>     (set-window-start-at-begin (point-min) (point))))
> (defun set-window-start-at-begin (beg end)
>   (set-window-start nil beg)
>   (unless (pos-visible-in-window-p end nil t)
>     (set-window-start-at-begin (+ beg (/ (- end beg) 2)) end)))
> (add-hook 'window-scroll-functions #'start-display-at-beginning-of-minibuffer)
> (add-hook 'post-command-hook #'start-display-at-beginning-of-minibuffer)

This might indeed give a reasonable behavior in practice, but I'm not
sure it's better than what we have now.
Also this has some problematic aspects:
- it focuses all its energy on showing the text before point, which is
  often the right choice, but not always.
- There's of course a risk of inf-loop if (set-window-start nil (1- end))
  leads to (pos-visible-in-window-p end nil t) returning nil.  How/when
  could this happen, I'm not completely sure, but it doesn't seem impossible.
- `window-scroll-functions` says explicitly:

      Warning: Do not use this feature to alter the way the window is
      scrolled.  It is not designed for that, and such use probably
      won’t work.

  Now, I know experience shows that it does work at least in some cases,
  but if so I think the code should come with a clear comment explaining
  why that warning doesn't apply here (and maybe the corresponding
  C code should also get a comment explaining the
  properties/invariants/berhaviors that it preserves and that make such
  uses work, so we don't break it by accident).

> 2. The only drawback of the above solution is that is is not possible to
>    display an ellipsis ("...") at the end of the completion candidates list,
>    to indicate that some completion candidates are not displayed.  It seems
>    to me that this is a minor limitation.

Actually, it seems like your code would allow to do that by running some
icomplete code at the end of your `set-window-start-at-begin` to
truncate the overlay's text according to where the text is truncated.

> 7. (add-hook 'post-command-hook 'start-display-at-beginning-of-minibuffer)
>    is necessary only with variable width faces, but it is does not harm to
>    use it with fixed width faces.

I don't understand why the kind of face in use would make a difference
w.r.t needing to use `post-command-hook`.


        Stefan

Re: A solution to display completion candidates after point in a minibuffer

[Gregory Heytings via Emacs development discussions.]

[-- Attachment #1: Type: text/plain, Size: 4758 bytes --]


Hi Stefan,

Thank you for your feedback!

>
> I think you should better define the problem you're trying to solve.
>

This problem is known to those who have been working on workarounds to 
circumvent it: displaying completion candidates in the minibuffer after 
prompt, without hiding the prompt.

>
> AFAIK the main issue ("when there are too many completion candidates, 
> the minibuffer prompt is hidden") has been solved by Eli
>

No it has not.  It has been solved in a partial way, which means that it 
will still be necessary to use workarounds to solve it in a clean, 
user-friendly way.  I sent a simple recipe on bug#43572, which 
demonstrates why Eli's solution is incomplete:

emacs -Q
(let (w bd)
   (setq w 60)
   (setq bd (concat (temporary-file-directory) (make-string w ?a) "/"))
   (dolist (d '("a" "b" "c" "d" "e")) (make-directory (concat bd d) t))
   (setq default-directory bd)
   (set-frame-height nil 20)
   (set-frame-width nil (+ (length bd) 10))
   (icomplete-mode)
   (setq icomplete-separator "\n")
   (call-interactively 'insert-file))

This recipe works with Emacs 24, 25, 26, 27 and 28, except that 
"(temporary-file-directory)" should be "temporary-file-directory" (without 
parentheses) in Emacs 24 and 25.

>
> [ I sightly tweaked your code ]
>

Thanks, your modified version works fine.

>
> Also this has some problematic aspects:
>
> - it focuses all its energy on showing the text before point, which is 
> often the right choice, but not always.
>

Indeed, that's not always the right choice, which is why this solution 
does this if, and only if, the buffer-local variable 
start-display-at-beginning-of-minibuffer has been set, in 
minibuffer-setup-hook.

>
> - There's of course a risk of inf-loop if (set-window-start nil (1- 
> end)) leads to (pos-visible-in-window-p end nil t) returning nil. 
> How/when could this happen, I'm not completely sure, but it doesn't seem 
> impossible.
>

In practice it's not possible, unless the width of the Emacs frame is so 
small that setting window-start near point (at point-1) would still leave 
point invisible.

>
> - `window-scroll-functions` says explicitly:
>
> Warning: Do not use this feature to alter the way the window is 
> scrolled.  It is not designed for that, and such use probably won’t 
> work.
>

Yes, I know ;-)

>
> Now, I know experience shows that it does work at least in some cases, 
> but if so I think the code should come with a clear comment explaining 
> why that warning doesn't apply here (and maybe the corresponding C code 
> should also get a comment explaining the 
> properties/invariants/berhaviors that it preserves and that make such 
> uses work, so we don't break it by accident).
>

This I cannot do, alas, I'm not an expert.  I tried this solution 
extensively, on different Emacs versions.  Perhaps there are cases where 
it does not work, but I doubt it.

>> 2. The only drawback of the above solution is that is is not possible 
>> to display an ellipsis ("...") at the end of the completion candidates 
>> list, to indicate that some completion candidates are not displayed. 
>> It seems to me that this is a minor limitation.
>
> Actually, it seems like your code would allow to do that by running some 
> icomplete code at the end of your `set-window-start-at-begin` to 
> truncate the overlay's text according to where the text is truncated.
>

That's possible, thanks for the idea, I'll try to see if this can be done. 
But I'm not sure, because modifying the contents of the buffer would mean 
that redisplay would be called again, which would again set window-start 
to another value.

>> 7. (add-hook 'post-command-hook 
>> 'start-display-at-beginning-of-minibuffer) is necessary only with 
>> variable width faces, but it is does not harm to use it with fixed 
>> width faces.
>
> I don't understand why the kind of face in use would make a difference 
> w.r.t needing to use `post-command-hook`.
>

I don't understand it either, alas.  An example, which does not work 
without start-display-at-beginning-of-minibuffer in post-command-hook with 
Emacs 26.3 (but works without it with Emacs 27.1):

emacs -Q
(let (w bd)
   (setq w 60)
   (setq bd (concat temporary-file-directory (make-string w ?a) "/"))
   (dolist (d '("a" "b" "c" "d" "e")) (make-directory (concat bd d) t))
   (setq default-directory bd)
   (set-frame-width nil 80)
   (set-frame-height nil 30)
   (icomplete-mode 1)
   (setq icomplete-separator "\n")
   (add-hook 'minibuffer-setup-hook (lambda () (buffer-face-mode t)))
   (set-face-attribute 'variable-pitch nil :height 1.2)
   (call-interactively 'insert-file)))

Re: A solution to display completion candidates after point in a minibuffer

[Stefan Monnier]

>> I think you should better define the problem you're trying to solve.
> This problem is known to those who have been working on workarounds to
> circumvent it: displaying completion candidates in the minibuffer after
> prompt, without hiding the prompt.

This description is not helpful because it is too vague.
E.g. it's not clear what you mean by "prompt".
is it really just the `minibuffer-prompt`?
If so, why do you specifically care about the prompt?
I usually find it more important to see the candidates than the prompt
(the prompt is important at the beginning, before you start writing
your answer, but after that it loses a lot of its importance).

Also, what do you mean by "without hiding the prompt".  There are cases
where there's no other option than to hide the prompt, so which cases
are you specifically interested in?  Also, what if some of the prompt is
visible but not all, is that a problem you're concerned with, or not?

> (let (w bd)
>   (setq w 60)
>   (setq bd (concat (temporary-file-directory) (make-string w ?a) "/"))

AKA

    (let ((w 60)
          (bd (expand-file-name (make-string w ?a) (temporary-file-directory))))

>   (dolist (d '("a" "b" "c" "d" "e")) (make-directory (concat bd d) t))
>   (setq default-directory bd)
>   (set-frame-height nil 20)
>   (set-frame-width nil (+ (length bd) 10))
>   (icomplete-mode)
>   (setq icomplete-separator "\n")
>   (call-interactively 'insert-file))

[ FWIW, I just tried it in my local Emacs where I replaced the ad-hoc
  `resize_mini_window` scrolling with the use of `scroll-conservatively`,
  and I get the behavior that you seem to prefer.  ]
[ Side note: I had sent a better recipe for that using a repetition of
  `foo/..` so as to avoid having to create any new files or
  directories.  ]

Thanks.  Indeed, this recipe shows a problem where the prompt is not
displayed at all right at the beginning of the interaction, so the
unwary user won't get to see what the prompt says without extra effort.
At the same time, the text that is displayed (i.e. without the prompt
but with one more line of candidates) is arguably a better choice when
the user knows what the prompt says (as would be the case for me
when I call `insert-file`).

So while I would tend to agree that showing the first line would
probably be better it's debatable.  At least, I don't think this
minor difference warrants the kind of code you're proposing.

>> Also this has some problematic aspects:
>> - it focuses all its energy on showing the text before point, which is
>> often the right choice, but not always.
> Indeed, that's not always the right choice, which is why this solution
> does this if, and only if, the buffer-local variable
> start-display-at-beginning-of-minibuffer has been set, in
> minibuffer-setup-hook.

But it depends on other factors than "displaying the minibuffer".
It can vary over the lifetime of the very same minibuffer.

>> - There's of course a risk of inf-loop if (set-window-start nil (1- end))
>> leads to (pos-visible-in-window-p end nil t) returning nil. How/when could
>> this happen, I'm not completely sure, but it doesn't seem impossible.
> In practice it's not possible, unless the width of the Emacs frame is so
> small that setting window-start near point (at point-1) would still leave
> point invisible.

It doesn't seem impossible even with a wide window.  I think risky
situations would include cases where point is right after a newline,
and/or where tall glyphs are involved (either via face settings, or
images, ...).

>> Now, I know experience shows that it does work at least in some cases, but
>> if so I think the code should come with a clear comment explaining why
>> that warning doesn't apply here (and maybe the corresponding C code should
>> also get a comment explaining the properties/invariants/berhaviors that it
>> preserves and that make such uses work, so we don't break it by accident).
> This I cannot do, alas, I'm not an expert.  I tried this solution
> extensively, on different Emacs versions.  Perhaps there are cases where it
> does not work, but I doubt it.

I'd be uneasy using such code without some vague understanding about
*why* it works.

>> Actually, it seems like your code would allow to do that by running some
>> icomplete code at the end of your `set-window-start-at-begin` to truncate
>> the overlay's text according to where the text is truncated.
> That's possible, thanks for the idea, I'll try to see if this can be
> done.  But I'm not sure, because modifying the contents of the buffer
> would mean that redisplay would be called again, which would again set
> window-start to another value.

Which is the kind of reason where we'd benefit from knowing *why* it works.

>>> 7. (add-hook 'post-command-hook
>>> 'start-display-at-beginning-of-minibuffer) is necessary only with
>>> variable width faces, but it is does not harm to use it with fixed
>>> width faces.
>> I don't understand why the kind of face in use would make a difference
>> w.r.t needing to use `post-command-hook`.
> I don't understand it either, alas.  An example, which does not work without
> start-display-at-beginning-of-minibuffer in post-command-hook with Emacs
> 26.3 (but works without it with Emacs 27.1):

Thanks for the example.  I think this highlights the need to better
understand how/why this works.


        Stefan

RE: A solution to display completion candidates after point in a minibuffer

[Drew Adams]

> > (let (w bd)
> >   (setq w 60)
> >   (setq bd (concat (temporary-file-directory) (make-string w ?a) "/"))
> 
> AKA
> 
>   (let ((w 60)
>         (bd (expand-file-name
>               (make-string w ?a)
>               (temporary-file-directory))))

let*

Re: A solution to display completion candidates after point in a minibuffer

[Gregory Heytings via Emacs development discussions.]

Hi Stefan,

Thanks again for your comments.

>
> Thanks.  Indeed, this recipe shows a problem where the prompt is not 
> displayed at all right at the beginning of the interaction, so the 
> unwary user won't get to see what the prompt says without extra effort. 
> At the same time, the text that is displayed (i.e. without the prompt 
> but with one more line of candidates) is arguably a better choice when 
> the user knows what the prompt says (as would be the case for me when I 
> call `insert-file`).
>

Completion candidates are a mere help for the user, they are not essential 
for them to know what to do.  The prompt (minibuffer-prompt) and their 
input are essential for them to know what to do.  Why would one more 
completion candidate on the last line of the minibuffer be more important, 
from the point of view of user-friendliness, than the minibuffer prompt 
and their input?

>
> So while I would tend to agree that showing the first line would 
> probably be better it's debatable.  At least, I don't think this minor 
> difference warrants the kind of code you're proposing.
>

It's what icomplete-vertical, ido-vertical and others try hard to do, so 
IMO that's a clear sign that this should not be debated.  See the ongoing 
discussion on making Emacs popular/accessible/user-friendly/...  That you 
or me, as experienced users, know that after hitting C-x i we are asking 
Emacs to insert a file does not mean that a beginner who would start Emacs 
in a too deep directory knows this or would be able to figure out what is 
happening.

>>> Also this has some problematic aspects: - it focuses all its energy on 
>>> showing the text before point, which is often the right choice, but 
>>> not always.
>>
>> Indeed, that's not always the right choice, which is why this solution 
>> does this if, and only if, the buffer-local variable 
>> start-display-at-beginning-of-minibuffer has been set, in 
>> minibuffer-setup-hook.
>
> But it depends on other factors than "displaying the minibuffer". It can 
> vary over the lifetime of the very same minibuffer.
>

Here I admit I have no idea what you mean.  AFAIU, each time the 
minibuffer is entered, a new "*Minibuf-N* buffer is created, so setting 
that variable would only have an effect on that specific minibuffer, while 
the user is completing a command or file name or ... in that specific 
minibuffer.

>>> - There's of course a risk of inf-loop if (set-window-start nil (1- 
>>> end)) leads to (pos-visible-in-window-p end nil t) returning nil. 
>>> How/when could this happen, I'm not completely sure, but it doesn't 
>>> seem impossible.
>>
>> In practice it's not possible, unless the width of the Emacs frame is 
>> so small that setting window-start near point (at point-1) would still 
>> leave point invisible.
>
> It doesn't seem impossible even with a wide window.  I think risky 
> situations would include cases where point is right after a newline, 
> and/or where tall glyphs are involved (either via face settings, or 
> images, ...).
>

If you think that's a real risk, it's easy to avoid this by enclosing the 
body of set-window-start-at-begin in a "(when (< (+ beg N) end)", with a 
suitable N; 1 or 2 should be okay.

>> This I cannot do, alas, I'm not an expert.  I tried this solution 
>> extensively, on different Emacs versions.  Perhaps there are cases 
>> where it does not work, but I doubt it.
>
> I'd be uneasy using such code without some vague understanding about 
> *why* it works.
>

The code of xdisp.c is rather intricate (to say the least), but if a vague 
understanding is enough, I can explain what I understood.  After 
resize_mini_window(), redisplay_window() is called, and its force_start 
part is executed, where run_window_scroll_functions() is called, which 
updates startp and therefore w->start.  This gives the user the 
possibility (and as far as I can see it is the only possibility for the 
user, with Lisp functions) to update window-start between 
resize_mini_window() and redisplay.  So set-window-start works, while 
other operations (such as an explicit scroll-up or scroll-down) might not.

>>> I don't understand why the kind of face in use would make a difference 
>>> w.r.t needing to use `post-command-hook`.
>>
>> I don't understand it either, alas.  An example, which does not work 
>> without start-display-at-beginning-of-minibuffer in post-command-hook 
>> with Emacs 26.3 (but works without it with Emacs 27.1):
>
> Thanks for the example.  I think this highlights the need to better 
> understand how/why this works.
>

In fact I think this example demonstrates a (minor) bug in Emacs, given 
that the exact same code gives a different behavior with different 
versions of Emacs.  I could only reproduce this bug with variable width 
faces, with which I guess that some rounding approximations happen here 
and there.

Re: A solution to display completion candidates after point in a minibuffer

[Gregory Heytings via Emacs development discussions.]

Hi Stefan,

Two comments I forgot to add in my previous mail:

>
> [ FWIW, I just tried it in my local Emacs where I replaced the ad-hoc 
> `resize_mini_window` scrolling with the use of `scroll-conservatively`, 
> and I get the behavior that you seem to prefer. ]
>

Yes, that's not surprising, what your code does is essentially (or very 
close to) what I suggested to do in bug#43519 and bug#43572, except that 
you do it unconditionally, when I suggested to do this only conditionally, 
when a variable is set.

>
> [ Side note: I had sent a better recipe for that using a repetition of 
> `foo/..` so as to avoid having to create any new files or directories. ]
>

No, your recipe won't work, because it is necessary for the recipe to work 
to have at least five completion candidates displayed, which is why the 
recipe creates five subdirectories.  Of course I could have created a 
recipe assuming that emacs -Q is started inside the Emacs source tree, but 
that would have been, at least for me, much harder to reproduce and test 
with various versions of Emacs.

Re: A solution to display completion candidates after point in a minibuffer

[Stefan Monnier]

>>>> Also this has some problematic aspects: - it focuses all its energy on
>>>> showing the text before point, which is often the right choice, but
>>>> not always.
>>> Indeed, that's not always the right choice, which is why this solution
>>> does this if, and only if, the buffer-local variable
>>> start-display-at-beginning-of-minibuffer has been set, in
>>> minibuffer-setup-hook.
>> But it depends on other factors than "displaying the minibuffer". It can
>> vary over the lifetime of the very same minibuffer.
> Here I admit I have no idea what you mean.

In your example recipe, the first line is hidden.  I agree that it's
probably a bad idea *when you enter the minibuffer*.  But this same
display is probably a better choice after the user read the prompt and
knows what's the current directory, at which point the list of remaining
completions is likely going to the main focus.

>>> This I cannot do, alas, I'm not an expert.  I tried this solution
>>> extensively, on different Emacs versions.  Perhaps there are cases where
>>> it does not work, but I doubt it.
>> I'd be uneasy using such code without some vague understanding about *why*
>> it works.
> The code of xdisp.c is rather intricate (to say the least), but if a vague
> understanding is enough, I can explain what I understood.
> After resize_mini_window(), redisplay_window() is called, and its
> force_start part is executed, where run_window_scroll_functions() is called,
> which updates startp and therefore w->start.  This gives the user the
> possibility (and as far as I can see it is the only possibility for the
> user, with Lisp functions) to update window-start between
> resize_mini_window() and redisplay.  So set-window-start works, while other
> operations (such as an explicit scroll-up or scroll-down) might not.

After `set-window-start`, the redisplay will be inevitably re-started,
which in turn might decide to scroll and thus
`run_window_scroll_functions`, etc...

IIRC the reason it won't scroll the second time around is because point
should be visible (and redisplay would only scroll in order to move
point within view).

>>>> I don't understand why the kind of face in use would make a difference
>>>> w.r.t needing to use `post-command-hook`.
>>>
>>> I don't understand it either, alas.  An example, which does not work
>>> without start-display-at-beginning-of-minibuffer in post-command-hook
>>> with Emacs 26.3 (but works without it with Emacs 27.1):
>>
>> Thanks for the example.  I think this highlights the need to better
>> understand how/why this works.
> In fact I think this example demonstrates a (minor) bug in Emacs, given that
> the exact same code gives a different behavior with different versions of
> Emacs.  I could only reproduce this bug with variable width faces, with
> which I guess that some rounding approximations happen here and there.

Maybe because from the redisplay's point of view, there is no
scrolling involved on the first redisplay of the minibuffer?


        Stefan

Re: A solution to display completion candidates after point in a minibuffer

[Gregory Heytings via Emacs development discussions.]

Hi Stefan,

>
> In your example recipe, the first line is hidden.  I agree that it's 
> probably a bad idea *when you enter the minibuffer*.  But this same 
> display is probably a better choice after the user read the prompt and 
> knows what's the current directory, at which point the list of remaining 
> completions is likely going to the main focus.
>

Thank you, now I see what you mean.  IMO (and I would be extremely 
surprised if I were the only one with that opinion) seeing the current 
directory disappearing is disturbing (and from a newcomer point of view: 
very disturbing), so the prompt an user input should always be displayed 
(unless the miniwindow is too small of course).

>
> After `set-window-start`, the redisplay will be inevitably re-started, 
> which in turn might decide to scroll and thus 
> `run_window_scroll_functions`, etc...
>
> IIRC the reason it won't scroll the second time around is because point 
> should be visible (and redisplay would only scroll in order to move 
> point within view).
>

I don't know, but I'm not sure about that.  If you (set-window-start nil 
1) unconditionally in window-scroll-functions, this setting will be obeyed 
by redisplay, even if point is not visible anymore.  Which, by the way, 
explains the need of a recursive call to set-window-start-at-begin.

Re: A solution to display completion candidates after point in a minibuffer

[Stefan Monnier]

>> [ FWIW, I just tried it in my local Emacs where I replaced the ad-hoc
>> `resize_mini_window` scrolling with the use of `scroll-conservatively`,
>> and I get the behavior that you seem to prefer. ]
> Yes, that's not surprising, what your code does is essentially (or very
> close to) what I suggested to do in bug#43519 and bug#43572,

Not really, because with my code, the window start is not explicitly set
to BOB unless the buffer's content is small enough to be completely
visible (the first version of my patch did, but not the current one).

> Thank you, now I see what you mean.  IMO (and I would be extremely
> surprised if I were the only one with that opinion) seeing the current
> directory disappearing is disturbing (and from a newcomer point of
> view: very disturbing), so the prompt an user input should always be
> displayed (unless the miniwindow is too small of course).

I agree it's suboptimal when entering the minibuffer.  OTOH it's
a perfectly acceptable behavior while editing the minibuffer's content
and can even be preferable in some cases to what you propose.

>> IIRC the reason it won't scroll the second time around is because point
>> should be visible (and redisplay would only scroll in order to move point
>> within view).
> I don't know, but I'm not sure about that.  If you (set-window-start nil 1)
> unconditionally in window-scroll-functions, this setting will be obeyed by
> redisplay, even if point is not visible anymore.

Oh, indeed, in that case it would move point instead.


        Stefan

Re: A solution to display completion candidates after point in a minibuffer

[Gregory Heytings via Emacs development discussions.]

>>> [ FWIW, I just tried it in my local Emacs where I replaced the ad-hoc 
>>> `resize_mini_window` scrolling with the use of 
>>> `scroll-conservatively`, and I get the behavior that you seem to 
>>> prefer. ]
>>
>> Yes, that's not surprising, what your code does is essentially (or very 
>> close to) what I suggested to do in bug#43519 and bug#43572,
>
> Not really, because with my code, the window start is not explicitly set 
> to BOB unless the buffer's content is small enough to be completely 
> visible (the first version of my patch did, but not the current one).
>

That's correct indeed, but setting window start explicitly in 
resize_mini_window() is only a hint for redisplay, that can ignore this 
setting in particular when when point would become invisible.  So in 
effect it's (very close to) what I suggested to do.

>> Thank you, now I see what you mean.  IMO (and I would be extremely 
>> surprised if I were the only one with that opinion) seeing the current 
>> directory disappearing is disturbing (and from a newcomer point of 
>> view: very disturbing), so the prompt an user input should always be 
>> displayed (unless the miniwindow is too small of course).
>
> I agree it's suboptimal when entering the minibuffer.  OTOH it's a 
> perfectly acceptable behavior while editing the minibuffer's content and 
> can even be preferable in some cases to what you propose.
>

I don't think so, IMO the minibuffer prompt should be visible at all 
times.  A rough equivalent of what you propose in a GUI would be, for 
example, for the Windows File Explorer or the macOS Finder to suddenly 
become fullscreen when the directory entered contains more files than it 
can display with its current size.  But I will not argue further.

Anyway, what you think could be a preferable behavior (showing the first 
line(s) when entering the minibuffer, and hiding it/them when the user has 
already entered some data) is easy to do with the minibuffer-local 
variable solution I propose.  It suffices to (setq 
start-display-at-beginning-of-minibuffer nil) at some point.  It could 
become a user preference, something like icomplete-always-display-prompt 
with a default value t.

>>> IIRC the reason it won't scroll the second time around is because 
>>> point should be visible (and redisplay would only scroll in order to 
>>> move point within view).
>>
>> I don't know, but I'm not sure about that.  If you (set-window-start 
>> nil 1) unconditionally in window-scroll-functions, this setting will be 
>> obeyed by redisplay, even if point is not visible anymore.
>
> Oh, indeed, in that case it would move point instead.
>

No, that's not what I meant.  In that case redisplay does not scroll and 
does not move point.  Point simply becomes invisible.  It becomes visible 
again after the next redisplay, a second or two later.

Re: A solution to display completion candidates after point in a minibuffer

[Eli Zaretskii]

> Date: Fri, 02 Oct 2020 21:30:15 +0000
> cc: emacs-devel@gnu.org
> From: Gregory Heytings via "Emacs development discussions." <emacs-devel@gnu.org>
> 
> In fact I think this example demonstrates a (minor) bug in Emacs, given 
> that the exact same code gives a different behavior with different 
> versions of Emacs.

The Emacs display code is stable, but it does evolve, and bugs are
being fixed regularly.  So it doesn't surprise me that something works
differently between two different versions, especially in code that
attempts to supplant the display engine instead of letting it do its
job.

That being said, writing code that works around redisplay does need a
good understanding of why things work like they do, and the above
might not be due to some bug that was fixed, but due to something else
entirely.  Waving the need to understand what goes on could well get
us in trouble down the road.

Re: A solution to display completion candidates after point in a minibuffer

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Fri, 02 Oct 2020 20:10:46 -0400
> Cc: emacs-devel@gnu.org
> 
> >> IIRC the reason it won't scroll the second time around is because point
> >> should be visible (and redisplay would only scroll in order to move point
> >> within view).
> > I don't know, but I'm not sure about that.  If you (set-window-start nil 1)
> > unconditionally in window-scroll-functions, this setting will be obeyed by
> > redisplay, even if point is not visible anymore.
> 
> Oh, indeed, in that case it would move point instead.

No, it could also reject the start-point and try anew, see try_window.

Re: A solution to display completion candidates after point in a minibuffer

[Eli Zaretskii]

> Date: Sat, 03 Oct 2020 06:59:16 +0000
> cc: emacs-devel@gnu.org
> From: Gregory Heytings via "Emacs development discussions." <emacs-devel@gnu.org>
> 
> >>> IIRC the reason it won't scroll the second time around is because 
> >>> point should be visible (and redisplay would only scroll in order to 
> >>> move point within view).
> >>
> >> I don't know, but I'm not sure about that.  If you (set-window-start 
> >> nil 1) unconditionally in window-scroll-functions, this setting will be 
> >> obeyed by redisplay, even if point is not visible anymore.
> >
> > Oh, indeed, in that case it would move point instead.
> 
> No, that's not what I meant.  In that case redisplay does not scroll and 
> does not move point.  Point simply becomes invisible.

No, redisplay will never accept a result that point is invisible.  If
it ever does display such situations, it's a bug that needs to be
fixed.

> It becomes visible again after the next redisplay, a second or two
> later.

If such a situation exists, please describe how to reproduce it,
because it's a bug we need to fix.  Emacs should never display a
window where point is not fully visible.

Re: A solution to display completion candidates after point in a minibuffer

[Gregory Heytings via Emacs development discussions.]

>
> I'd be uneasy using such code without some vague understanding about 
> *why* it works.
>

I studied this in a bit more detail.

What happens in the normal case (with icomplete-mode, icomplete-separator 
set to "\n", and window-scroll-functions set to nil) is the following:

1. redisplay_internal() calls resize_mini_window() and 
redisplay_windows(),

2. redisplay_windows() calls redisplay_window_0(), which calls 
redisplay_window(), and the "recenter" part of redisplay_window() is 
executed.

This happens twice (for each window in the frame) before 
redisplay_internal() returns.  It happens twice because FRAME_GARBAGED_P, 
because the windows have been resized, which means that the "goto retry" 
jump is executed.  This means that run_window_scroll_functions() is called 
twice (which has of course no effect if window-scroll-functions is nil).

When window-scroll-functions is set to 
start-display-at-beginning-of-minibuffer, the same happens, except of 
course that start-display-at-beginning-of-minibuffer is executed twice. 
The first time the "recenter" part of redisplay_window() is executed, 
start-display-at-beginning-of-minibuffer sets w->force_start and w->start. 
Then resize_mini_window() sets w->start again, but redisplay_window() is 
called again, and this time it is its "force_start" part that is executed, 
which calls run_window_scroll_functions(), after which try_window() 
updates w->start to the window-start value set in window-scroll-functions.

Re: A solution to display completion candidates after point in a minibuffer

[Gregory Heytings via Emacs development discussions.]

>>>> I don't know, but I'm not sure about that.  If you (set-window-start 
>>>> nil 1) unconditionally in window-scroll-functions, this setting will 
>>>> be obeyed by redisplay, even if point is not visible anymore.
>>>
>>> Oh, indeed, in that case it would move point instead.
>>
>> No, that's not what I meant.  In that case redisplay does not scroll 
>> and does not move point.  Point simply becomes invisible.
>
> No, redisplay will never accept a result that point is invisible.  If it 
> ever does display such situations, it's a bug that needs to be fixed.
>

Never say never...

>> It becomes visible again after the next redisplay, a second or two 
>> later.
>
> If such a situation exists, please describe how to reproduce it, because 
> it's a bug we need to fix.  Emacs should never display a window where 
> point is not fully visible.
>

Here's a recipe:

emacs -Q
(defun recipe ()
   (interactive)
   (set-frame-width nil 80)
   (set-frame-height nil 30)
   (switch-to-buffer (get-buffer-create "*Recipe*"))
   (goto-char 1)
   (let ((i 0)) (while (< i 60) (setq i (1+ i)) (insert (format "L%d\n" i))))
   (goto-char 148)
   (sit-for 0.25)
   (add-hook 'window-scroll-functions (lambda (&rest args) (set-window-start nil 1)))
   (forward-line 13))
M-x recipe
press C-n

You will see the cursor on the first line, after "L1", but in fact point 
is at that moment not visible anymore, it is still on "L54".  A second or 
two later, point has actually moved, and you will see the cursor blinking 
on "L14".  This works at least with Emacs 23, 24, 25, 26, 27 and 28.

The bug is more visible with Emacs 23 and 24, because you don't see the 
cursor on the first line after pressing C-n, you see a window without any 
cursor during a second or two.  With Emacs 25, 26, 27 and 28 the cursor 
after "L1" does not correspond to point (and as far as I understand does 
not correspond to anything).

Re: A solution to display completion candidates after point in a minibuffer

[Eli Zaretskii]

> Date: Sun, 04 Oct 2020 16:11:06 +0000
> From: Gregory Heytings <ghe@sdf.org>
> cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> > No, redisplay will never accept a result that point is invisible.  If it 
> > ever does display such situations, it's a bug that needs to be fixed.
> 
> Never say never...
> 
> >> It becomes visible again after the next redisplay, a second or two 
> >> later.
> >
> > If such a situation exists, please describe how to reproduce it, because 
> > it's a bug we need to fix.  Emacs should never display a window where 
> > point is not fully visible.
> >
> 
> Here's a recipe:
> 
> emacs -Q
> (defun recipe ()
>    (interactive)
>    (set-frame-width nil 80)
>    (set-frame-height nil 30)
>    (switch-to-buffer (get-buffer-create "*Recipe*"))
>    (goto-char 1)
>    (let ((i 0)) (while (< i 60) (setq i (1+ i)) (insert (format "L%d\n" i))))
>    (goto-char 148)
>    (sit-for 0.25)
>    (add-hook 'window-scroll-functions (lambda (&rest args) (set-window-start nil 1)))
>    (forward-line 13))
> M-x recipe
> press C-n

AFAIU, this is exactly why we say not to use window-scroll-functions
for this purpose.  IOW, there is indeed a bug here, and the bug is in
the Lisp code you presented: window-scroll-functions are not meant to
be used to affect redisplay, they are meant to inform Lisp programs
about what redisplay is about to do with a window.

Re: A solution to display completion candidates after point in a minibuffer

[Gregory Heytings via Emacs development discussions.]

>>>> No, that's not what I meant.  In that case redisplay does not scroll 
>>>> and does not move point.  Point simply becomes invisible.
>>>
>>> No, redisplay will never accept a result that point is invisible.  If 
>>> it ever does display such situations, it's a bug that needs to be 
>>> fixed.
>>
>> Never say never...
>>
>>>> It becomes visible again after the next redisplay, a second or two 
>>>> later.
>>>
>>> If such a situation exists, please describe how to reproduce it, 
>>> because it's a bug we need to fix.  Emacs should never display a 
>>> window where point is not fully visible.
>>
>> Here's a recipe:
>>
>> emacs -Q
>> (defun recipe ()
>>    (interactive)
>>    (set-frame-width nil 80)
>>    (set-frame-height nil 30)
>>    (switch-to-buffer (get-buffer-create "*Recipe*"))
>>    (goto-char 1)
>>    (let ((i 0)) (while (< i 60) (setq i (1+ i)) (insert (format "L%d\n" i))))
>>    (goto-char 148)
>>    (sit-for 0.25)
>>    (add-hook 'window-scroll-functions (lambda (&rest args) (set-window-start nil 1)))
>>    (forward-line 13))
>> M-x recipe
>> press C-n
>
> AFAIU, this is exactly why we say not to use window-scroll-functions for 
> this purpose.  IOW, there is indeed a bug here, and the bug is in the 
> Lisp code you presented: window-scroll-functions are not meant to be 
> used to affect redisplay, they are meant to inform Lisp programs about 
> what redisplay is about to do with a window.
>

I did not say that there was a bug in redisplay here, I explained what 
happens, and you said there was a bug, that redisplay should never do 
this, and asked me to write a recipe, which I did.

Obviously, I do not do what I did above, to demonstrate what you said was 
a bug, in my code.

Re: A solution to display completion candidates after point in a minibuffer

[Eli Zaretskii]

> Date: Sun, 04 Oct 2020 16:52:10 +0000
> From: Gregory Heytings <ghe@sdf.org>
> cc: monnier@iro.umontreal.ca, emacs-devel@gnu.org
> 
> > AFAIU, this is exactly why we say not to use window-scroll-functions for 
> > this purpose.  IOW, there is indeed a bug here, and the bug is in the 
> > Lisp code you presented: window-scroll-functions are not meant to be 
> > used to affect redisplay, they are meant to inform Lisp programs about 
> > what redisplay is about to do with a window.
> 
> I did not say that there was a bug in redisplay here, I explained what 
> happens, and you said there was a bug, that redisplay should never do 
> this, and asked me to write a recipe, which I did.

Thank you, I was under the impression that redisplay did something like
that by itself, or when the various hooks are used as intended.  I'm
relieved to learn that is not so.