Updating *Completions* as you type

Updating *Completions* as you type

[sbaugh]

It would be nice if there was a built-in customization which caused
*Completions* to update as you type, as long as that buffer is visible.
I imagine such a request has been made before, so what is the obstacle
to adding it?

I would like to figure out a solution which everyone is happy with, and
then I would be happy to implement it.  It seems to me that it's just a
matter of, after each keystroke, triggering (for minibuffer completion)
minibuffer-completion-help or (for buffer completion) some new function
to populate *Completions* with all-completions output.  This could (I
guess) be done with after-change-functions, although maybe others have a
better idea.

(Btw, if we had this behavior, it also seems like it would help with
another long-time request: asynchronous completion support.  A
programmed completion table could internally do something asynchronous
and stateful, and accumulate results over time, and return more and more
results each time Emacs calls all-completions/try-completion.  If Emacs
automatically called all-completions with such a programmed completion
table, the resulting behavior would be an acceptable approximation of
asynchronous completion, without having to complicate the programmed
completion API.)

Re: Updating *Completions* as you type

[Eli Zaretskii]

> From: sbaugh@catern.com
> Date: Thu, 12 Oct 2023 19:53:53 -0400
> 
> 
> It would be nice if there was a built-in customization which caused
> *Completions* to update as you type, as long as that buffer is visible.
> I imagine such a request has been made before, so what is the obstacle
> to adding it?

I think we have alternative completion schemes, like ido.el and
others, which update the list of the completion candidates as you
type, albeit not in the *Completions* buffer.  Why would we need yet
another knob in the default completion scheme?

> (Btw, if we had this behavior, it also seems like it would help with
> another long-time request: asynchronous completion support.  A
> programmed completion table could internally do something asynchronous
> and stateful, and accumulate results over time, and return more and more
> results each time Emacs calls all-completions/try-completion.  If Emacs
> automatically called all-completions with such a programmed completion
> table, the resulting behavior would be an acceptable approximation of
> asynchronous completion, without having to complicate the programmed
> completion API.)

IIUC what you mean by "asynchronous completion", it is a terrible
idea.  How can anyone type text without getting immediate feedback for
what he/she typed?  When I sometimes see this (due to network delays
or system load or whatever), it is a terrible UX.

Re: Updating *Completions* as you type

[Spencer Baugh]

Eli Zaretskii <eliz@gnu.org> writes:
>> From: sbaugh@catern.com
>> Date: Thu, 12 Oct 2023 19:53:53 -0400
>> 
>> 
>> It would be nice if there was a built-in customization which caused
>> *Completions* to update as you type, as long as that buffer is visible.
>> I imagine such a request has been made before, so what is the obstacle
>> to adding it?
>
> I think we have alternative completion schemes, like ido.el and
> others, which update the list of the completion candidates as you
> type, albeit not in the *Completions* buffer.  Why would we need yet
> another knob in the default completion scheme?

ido-mode only works for files and buffers, not all minibuffer
completion, and is also unmaintained and sometimes buggy besides that.

icomplete-mode (and its derivatives) doesn't work for in-buffer
completion.  Also, it is buggier than the default completion scheme.

>> (Btw, if we had this behavior, it also seems like it would help with
>> another long-time request: asynchronous completion support.  A
>> programmed completion table could internally do something asynchronous
>> and stateful, and accumulate results over time, and return more and more
>> results each time Emacs calls all-completions/try-completion.  If Emacs
>> automatically called all-completions with such a programmed completion
>> table, the resulting behavior would be an acceptable approximation of
>> asynchronous completion, without having to complicate the programmed
>> completion API.)
>
> IIUC what you mean by "asynchronous completion", it is a terrible
> idea.  How can anyone type text without getting immediate feedback for
> what he/she typed?  When I sometimes see this (due to network delays
> or system load or whatever), it is a terrible UX.

By asynchronous completion, I just mean being able to type, and trigger
completion, and have completions appear incrementally as they are
computed, without interfering with you continuing to type.  I don't
think that stops immediate feedback for typing.

Re: Updating *Completions* as you type

[Eli Zaretskii]

> From: Spencer Baugh <sbaugh@janestreet.com>
> Date: Fri, 13 Oct 2023 14:01:38 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> >> From: sbaugh@catern.com
> >> Date: Thu, 12 Oct 2023 19:53:53 -0400
> >> 
> >> 
> >> It would be nice if there was a built-in customization which caused
> >> *Completions* to update as you type, as long as that buffer is visible.
> >> I imagine such a request has been made before, so what is the obstacle
> >> to adding it?
> >
> > I think we have alternative completion schemes, like ido.el and
> > others, which update the list of the completion candidates as you
> > type, albeit not in the *Completions* buffer.  Why would we need yet
> > another knob in the default completion scheme?
> 
> ido-mode only works for files and buffers, not all minibuffer
> completion, and is also unmaintained and sometimes buggy besides that.
> 
> icomplete-mode (and its derivatives) doesn't work for in-buffer
> completion.

What do you mean by "in-buffer completion"?

> Also, it is buggier than the default completion scheme.

Then let's fix those bugs before we consider adding yet another
completion scheme (with its own bugs).

Does anyone else here thinks we need yet in core another completion
feature in addition to what we have already (which, btw, includes
fido-mode as well)?

> >> (Btw, if we had this behavior, it also seems like it would help with
> >> another long-time request: asynchronous completion support.  A
> >> programmed completion table could internally do something asynchronous
> >> and stateful, and accumulate results over time, and return more and more
> >> results each time Emacs calls all-completions/try-completion.  If Emacs
> >> automatically called all-completions with such a programmed completion
> >> table, the resulting behavior would be an acceptable approximation of
> >> asynchronous completion, without having to complicate the programmed
> >> completion API.)
> >
> > IIUC what you mean by "asynchronous completion", it is a terrible
> > idea.  How can anyone type text without getting immediate feedback for
> > what he/she typed?  When I sometimes see this (due to network delays
> > or system load or whatever), it is a terrible UX.
> 
> By asynchronous completion, I just mean being able to type, and trigger
> completion, and have completions appear incrementally as they are
> computed, without interfering with you continuing to type.  I don't
> think that stops immediate feedback for typing.

If it's asynchronous, then how can we make sure the feedback is
immediate?

Re: Updating *Completions* as you type

[Björn Bidar]

Eli Zaretskii <eliz@gnu.org> writes:

>> > IIUC what you mean by "asynchronous completion", it is a terrible
>> > idea.  How can anyone type text without getting immediate feedback for
>> > what he/she typed?  When I sometimes see this (due to network delays
>> > or system load or whatever), it is a terrible UX.
>> 
>> By asynchronous completion, I just mean being able to type, and trigger
>> completion, and have completions appear incrementally as they are
>> computed, without interfering with you continuing to type.  I don't
>> think that stops immediate feedback for typing.
>
> If it's asynchronous, then how can we make sure the feedback is
> immediate?

The feedback updates while typing but doesn't show until the delay set
by the user has passed to show the suggestions again.

If the feedback hasn't updated by the time the user stops typing then
the old state is shown similar to how  the feedback is when the
completion is synchronous.
I most cases I think the feedback updates as fast as the major mode can
update the buffer to pass it e.g. to an internal function or language
server/or alike.
In cases of external programs parsing the just updated buffer the
completion can be nearly asynchronous already.

Re: Updating *Completions* as you type

[Eli Zaretskii]

> From: Björn Bidar <bjorn.bidar@thaodan.de>
> Cc: Spencer Baugh <sbaugh@janestreet.com>,  emacs-devel@gnu.org
> Date: Sat, 14 Oct 2023 22:26:52 +0300
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> > IIUC what you mean by "asynchronous completion", it is a terrible
> >> > idea.  How can anyone type text without getting immediate feedback for
> >> > what he/she typed?  When I sometimes see this (due to network delays
> >> > or system load or whatever), it is a terrible UX.
> >> 
> >> By asynchronous completion, I just mean being able to type, and trigger
> >> completion, and have completions appear incrementally as they are
> >> computed, without interfering with you continuing to type.  I don't
> >> think that stops immediate feedback for typing.
> >
> > If it's asynchronous, then how can we make sure the feedback is
> > immediate?
> 
> The feedback updates while typing but doesn't show until the delay set
> by the user has passed to show the suggestions again.

This sentence seems to contradict itself: what is the meaning of
"feedback updates" if the feedback "doesn't show"?

> If the feedback hasn't updated by the time the user stops typing then
> the old state is shown similar to how  the feedback is when the
> completion is synchronous.

I don't understand this, either.  Please elaborate, or maybe show an
example.

Re: Updating *Completions* as you type

[Juri Linkov]

> icomplete-mode (and its derivatives) doesn't work for in-buffer
> completion.  Also, it is buggier than the default completion scheme.

Actually, now icomplete-mode supports in-buffer completion quite well.

> By asynchronous completion, I just mean being able to type, and trigger
> completion, and have completions appear incrementally as they are
> computed, without interfering with you continuing to type.  I don't
> think that stops immediate feedback for typing.

I believe this is achievable by just wrapping code with while-no-input.

Re: Updating *Completions* as you type

[sbaugh]

Juri Linkov <juri@linkov.net> writes:

>> icomplete-mode (and its derivatives) doesn't work for in-buffer
>> completion.  Also, it is buggier than the default completion scheme.
>
> Actually, now icomplete-mode supports in-buffer completion quite well.

Oh, neat!  I knew there was icomplete-in-buffer but I see that it's been
much improved recently.

>> By asynchronous completion, I just mean being able to type, and trigger
>> completion, and have completions appear incrementally as they are
>> computed, without interfering with you continuing to type.  I don't
>> think that stops immediate feedback for typing.
>
> I believe this is achievable by just wrapping code with while-no-input.

Indeed.

Re: Updating *Completions* as you type

[Dmitry Gutov]

On 14/10/2023 19:51, Juri Linkov wrote:
>> icomplete-mode (and its derivatives) doesn't work for in-buffer
>> completion.  Also, it is buggier than the default completion scheme.
> Actually, now icomplete-mode supports in-buffer completion quite well.
> 
>> By asynchronous completion, I just mean being able to type, and trigger
>> completion, and have completions appear incrementally as they are
>> computed, without interfering with you continuing to type.  I don't
>> think that stops immediate feedback for typing.
> I believe this is achievable by just wrapping code with while-no-input.
> 

icomplete-exhibit uses while-no-input, FWIW.

Re: Updating *Completions* as you type

[Juri Linkov]

[-- Attachment #1: Type: text/plain, Size: 1389 bytes --]

> It would be nice if there was a built-in customization which caused
> *Completions* to update as you type, as long as that buffer is visible.
> I imagine such a request has been made before, so what is the obstacle
> to adding it?

I don't remember what was the obstacle, but here is the previous patch
that implements the behavior of zsh and is based on icomplete-mode.

> I would like to figure out a solution which everyone is happy with, and
> then I would be happy to implement it.  It seems to me that it's just a
> matter of, after each keystroke, triggering (for minibuffer completion)
> minibuffer-completion-help or (for buffer completion) some new function
> to populate *Completions* with all-completions output.

There is one difference between icomplete-mode and zcomplete-mode:
the former uses completion-all-sorted-completions where the
recently used items are at the top of the completion list.
Whereas the latter uses the same alphabetical sorting as TAB
shows in the *Completions* buffer.  Maybe a new function
should allow any sorting order?

You are welcome to implement such a new function
to populate *Completions* with all-completions output.
Currently I have no idea how to do this.

> This could (I guess) be done with after-change-functions, although
> maybe others have a better idea.

icomplete-mode uses post-command-hook, so zcomplete-mode does the same.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: zcomplete.patch --]
[-- Type: text/x-diff, Size: 13430 bytes --]

diff --git a/lisp/zcomplete.el b/lisp/zcomplete.el
new file mode 100644
index 00000000000..75a40c0afd3
--- /dev/null
+++ b/lisp/zcomplete.el
@@ -0,0 +1,317 @@
+;;; zcomplete.el --- zsh-like minibuffer completion based on icomplete -*- lexical-binding: t -*-
+
+;; Copyright (C) 2022 Free Software Foundation, Inc.
+
+;; Author: Juri Linkov <juri@linkov.net>
+;; Keywords: completion
+;; Maintainer: emacs-devel@gnu.org
+
+;; This file is part of GNU Emacs.
+
+;; GNU Emacs is free software: you can redistribute it and/or modify
+;; it under the terms of the GNU General Public License as published by
+;; the Free Software Foundation, either version 3 of the License, or
+;; (at your option) any later version.
+
+;; GNU Emacs is distributed in the hope that it will be useful,
+;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;; GNU General Public License for more details.
+
+;; You should have received a copy of the GNU General Public License
+;; along with GNU Emacs.  If not, see <https://www.gnu.org/licenses/>.
+
+;;; Commentary:
+
+;; Like `icomplete' but provides feedback in the *Completions* window
+;; instead of the minibuffer.
+
+;;; Code:
+
+(defgroup zcomplete nil
+  "Show completions dynamically in *Completions* window from minibuffer."
+  :prefix "zcomplete-"
+  :link '(info-link "(emacs)Zcomplete")
+  :group 'minibuffer)
+
+(defcustom zcomplete-show-matches-on-no-input nil
+  "When non-nil, show completions when first prompting for input.
+This means to show completions even when the current minibuffer contents
+is the same as was the initial input after minibuffer activation.
+This also means that if you traverse the list of completions with
+commands and just hit RET without typing any characters,
+the match under point will be chosen instead of the default."
+  :type 'boolean)
+
+(defcustom zcomplete-with-completion-tables t
+  "Specialized completion tables with which zcomplete should operate.
+If this is t, zcomplete operates on all tables.
+Otherwise this should be a list of the completion tables (e.g.,
+`internal-complete-buffer') on which zcomplete should operate."
+  :type '(choice (const :tag "All" t)
+                 (repeat function)))
+
+(defcustom zcomplete-compute-delay .15
+  "Completions-computation stall, used only with large-number completions.
+See `zcomplete-delay-completions-threshold'."
+  :type 'number)
+
+(defcustom zcomplete-delay-completions-threshold 400
+  "Pending-completions number over which to apply `zcomplete-compute-delay'."
+  :type 'integer)
+
+(defcustom zcomplete-max-delay-chars 2
+  "Maximum number of initial chars to apply `zcomplete-compute-delay'."
+  :type 'integer)
+
+(defcustom zcomplete-minibuffer-setup-hook nil
+  "Zcomplete-specific customization of minibuffer setup.
+This hook is run during minibuffer setup if Zcomplete is active."
+  :type 'hook)
+
+\f
+(defvar zcomplete--initial-input nil
+  "Initial input in the minibuffer when `zcomplete-mode' was activated.
+Used to implement the option `zcomplete-show-matches-on-no-input'.")
+
+(defvar zcomplete--previous-input nil
+  "Previous input in the minibuffer before editing it.
+Used to optimize `zcomplete-exhibit' to not be fired while
+moving point in the minibuffer.")
+
+(defun zcomplete-post-command-hook ()
+  (let ((non-essential t)) ;E.g. don't prompt for password!
+    (unless (memq this-command '( previous-history-element next-history-element
+                                  previous-line-or-history-element
+                                  next-line-or-history-element))
+      (zcomplete-exhibit))))
+
+(defcustom zcomplete-auto-exhibit 'visible
+  "Non-nil means to use pop up completions on minibuffer edit."
+  :type '(choice (const :tag "Don't auto pop up completions" nil)
+                 (const :tag "Pop up completions window" t)
+                 (const :tag "Update completions window only when visible"
+                        visible)))
+
+(defcustom zcomplete-arrows 'visible
+  "Non-nil means to use arrows to browse completions from the minibuffer."
+  :type '(choice (const :tag "Don't use arrows" nil)
+                 (const :tag "Use arrows" t)
+                 (const :tag "Use arrows when completions window is visible"
+                        visible)))
+
+(defun zcomplete-visible ()
+  (get-buffer-window "*Completions*" 0))
+
+(defun zcomplete-bind-arrows (binding &optional horizontal)
+  `(menu-item
+    "" ,binding
+    :filter ,(lambda (cmd)
+               (when (or (eq zcomplete-arrows t)
+                         (and (eq zcomplete-arrows 'visible)
+                              (zcomplete-visible)
+                              (or (not horizontal)
+                                  (eq completions-format 'one-column))))
+                 cmd))))
+
+(defun zcomplete-bind-visible (binding)
+  `(menu-item
+    "" ,binding
+    :filter ,(lambda (cmd)
+               (when (zcomplete-visible)
+                 cmd))))
+
+(defvar-keymap zcomplete-minibuffer-mode-map ;; zcomplete-minibuffer-map
+  :doc "Keymap used by `zcomplete-mode' in the minibuffer."
+
+  "<remap> <minibuffer-complete-and-exit>" (zcomplete-bind-visible #'zcomplete-ret)
+  "<remap> <minibuffer-keyboard-quit>"     (zcomplete-bind-visible #'zcomplete-quit)
+  "<remap> <abort-minibuffers>"            (zcomplete-bind-visible #'zcomplete-quit)
+
+  "<left>"    (zcomplete-bind-arrows #'zcomplete-previous-completion t)
+  "<right>"   (zcomplete-bind-arrows #'zcomplete-next-completion t)
+  "<up>"      (zcomplete-bind-arrows #'zcomplete-previous-line-completion)
+  "<down>"    (zcomplete-bind-arrows #'zcomplete-next-line-completion)
+  "<home>"    (zcomplete-bind-arrows #'zcomplete-first-completion)
+  "<end>"     (zcomplete-bind-arrows #'zcomplete-last-completion)
+  "<next>"    (zcomplete-bind-arrows #'scroll-other-window)
+  "<prior>"   (zcomplete-bind-arrows #'scroll-other-window-down))
+
+(defun zcomplete-ret (&optional no-exit no-quit)
+  "Choose the completion from the minibuffer in its completions window."
+  (interactive "P")
+  (condition-case nil
+      (minibuffer-choose-completion no-exit no-quit)
+    (error (minibuffer-complete-and-exit))))
+
+(defun zcomplete-quit ()
+  "Exit minibuffer for zcomplete."
+  (interactive)
+  (minibuffer-hide-completions))
+
+(defcustom zcomplete-auto-choose nil
+  "Non-nil means to automatically insert completions to the minibuffer.
+It affects the variable `minibuffer-completion-auto-choose'.
+This variable is usable only when `zcomplete-auto-exhibit' is nil."
+  :type 'boolean)
+
+(defun zcomplete-next-completion (&optional n)
+  "Run `minibuffer-next-completion' without auto choosing."
+  (interactive "p")
+  (let ((minibuffer-completion-auto-choose zcomplete-auto-choose))
+    (minibuffer-next-completion n)))
+
+(defun zcomplete-previous-completion (&optional n)
+  "Run `minibuffer-previous-completion' without auto choosing."
+  (interactive "p")
+  (let ((minibuffer-completion-auto-choose zcomplete-auto-choose))
+    (minibuffer-previous-completion n)))
+
+(defun zcomplete-next-line-completion (&optional n)
+  "Run `minibuffer-next-line-completion' without auto choosing."
+  (interactive "p")
+  (let ((minibuffer-completion-auto-choose zcomplete-auto-choose))
+    (minibuffer-next-line-completion n)))
+
+(defun zcomplete-previous-line-completion (&optional n)
+  "Run `minibuffer-previous-line-completion' without auto choosing."
+  (interactive "p")
+  (let ((minibuffer-completion-auto-choose zcomplete-auto-choose))
+    (minibuffer-previous-line-completion n)))
+
+(defun zcomplete-first-completion ()
+  "Run `first-completion' from the minibuffer in its completions window."
+  (interactive)
+  (with-minibuffer-completions-window
+    (when completions-highlight-face
+      (setq-local cursor-face-highlight-nonselected-window t))
+    (let ((minibuffer-completion-auto-choose zcomplete-auto-choose))
+      (first-completion))))
+
+(defun zcomplete-last-completion ()
+  "Run `last-completion' from the minibuffer in its completions window."
+  (interactive)
+  (with-minibuffer-completions-window
+    (when completions-highlight-face
+      (setq-local cursor-face-highlight-nonselected-window t))
+    (let ((minibuffer-completion-auto-choose zcomplete-auto-choose))
+      (last-completion))))
+
+\f
+;;;###autoload
+(define-minor-mode zcomplete-mode
+  "Toggle incremental minibuffer completion (Zcomplete mode).
+
+When this global minor mode is enabled, typing in the minibuffer
+continuously displays a list of possible completions that match
+the string you have typed.  The list of completions is displayed
+in the *Completions* window.
+
+For more information, see Info node `(emacs)Zcomplete'.
+For options you can set, `\\[customize-group] zcomplete'.
+
+You can use the following key bindings to navigate and select
+completions:
+
+\\{zcomplete-minibuffer-map}"
+  :global t :group 'zcomplete
+  (remove-hook 'minibuffer-setup-hook #'zcomplete-minibuffer-setup)
+  (when zcomplete-mode
+    (add-hook 'minibuffer-setup-hook #'zcomplete-minibuffer-setup)
+    ;; (setq-default completion-show-help nil
+    ;;               completions-header-format nil)
+    ))
+
+(define-minor-mode zcomplete-minibuffer-mode
+  "Enable arrows in the minibuffer.
+The only purpose of this mode is to activate
+`zcomplete-minibuffer-mode-map' in the minibuffer."
+  :global nil)
+
+(defun zcomplete--completion-table ()
+  (if (window-minibuffer-p) minibuffer-completion-table
+    (or (nth 2 completion-in-region--data)
+        (message "In %S (w=%S): %S"
+                 (current-buffer) (selected-window) (window-minibuffer-p)))))
+(defun zcomplete--field-string ()
+  (if (window-minibuffer-p)
+      (minibuffer-contents)
+    (buffer-substring-no-properties
+     (nth 0 completion-in-region--data)
+     (nth 1 completion-in-region--data))))
+(defun zcomplete--field-beg ()
+  (if (window-minibuffer-p) (minibuffer-prompt-end)
+    (nth 0 completion-in-region--data)))
+(defun zcomplete--field-end ()
+  (if (window-minibuffer-p) (point-max)
+    (nth 1 completion-in-region--data)))
+
+(defun zcomplete-simple-completing-p ()
+  "Non-nil if current window is a minibuffer that's doing simple completion."
+  (unless executing-kbd-macro
+    (let ((table (zcomplete--completion-table)))
+      (and table
+           (or (not (functionp table))
+               (eq zcomplete-with-completion-tables t)
+               (member table zcomplete-with-completion-tables))))))
+
+(defun zcomplete-minibuffer-setup ()
+  "Run in minibuffer on activation to establish incremental completion.
+Usually run by inclusion in `minibuffer-setup-hook'."
+  (when zcomplete-mode
+    (setq-local zcomplete--initial-input (zcomplete--field-string))
+    (setq-local zcomplete--previous-input nil)
+    (zcomplete-minibuffer-mode 1)
+    (add-hook 'post-command-hook #'zcomplete-post-command-hook nil t)
+    (zcomplete-exhibit)
+    (run-hooks 'zcomplete-minibuffer-setup-hook)))
+
+(defun zcomplete-exhibit ()
+  "Update zcomplete completions display.
+Should be run via minibuffer `post-command-hook'.
+See `zcomplete-mode' and `minibuffer-setup-hook'."
+  (when (and zcomplete-mode
+             (zcomplete-simple-completing-p)) ;Shouldn't be necessary.
+    (when (and (or zcomplete-show-matches-on-no-input
+                   (not (equal (zcomplete--field-string)
+                               zcomplete--initial-input)))
+               (not (equal (zcomplete--field-string)
+                           zcomplete--previous-input))
+               (or (eq zcomplete-auto-exhibit t)
+                   (and (eq zcomplete-auto-exhibit 'visible)
+                        (zcomplete-visible)))
+               (or
+                ;; Don't bother with delay after certain number of chars:
+                (> (- (point) (zcomplete--field-beg))
+                   zcomplete-max-delay-chars)
+                ;; Don't delay if the completions are known.
+                completion-all-sorted-completions
+                ;; Don't delay if alternatives number is small enough:
+                (and (sequencep (zcomplete--completion-table))
+                     (< (length (zcomplete--completion-table))
+                        zcomplete-delay-completions-threshold))
+                ;; Delay - give some grace time for next keystroke, before
+                ;; embarking on computing completions:
+                (sit-for zcomplete-compute-delay)))
+      (save-excursion
+        (goto-char (point-max))
+        (setq-local zcomplete--previous-input (zcomplete--field-string))
+        (minibuffer-completion-help)
+        (unless zcomplete-auto-exhibit
+          (zcomplete-first-completion))))))
+
+(let ((keymap completion-in-region-mode-map))
+  (keymap-set keymap "<left>"  #'zcomplete-previous-completion)
+  (keymap-set keymap "<right>" #'zcomplete-next-completion)
+  (keymap-set keymap "<up>"    #'zcomplete-previous-line-completion)
+  (keymap-set keymap "<down>"  #'zcomplete-next-line-completion)
+  (keymap-set keymap "<home>"  #'zcomplete-first-completion)
+  (keymap-set keymap "<end>"   #'zcomplete-last-completion)
+  (keymap-set keymap "<next>"  #'scroll-other-window)
+  (keymap-set keymap "<prior>" #'scroll-other-window-down)
+  (keymap-set keymap "RET"     #'zcomplete-ret))
+
+\f
+(provide 'zcomplete)
+
+;;; zcomplete.el ends here

Re: Updating *Completions* as you type

[Spencer Baugh]

Juri Linkov <juri@linkov.net> writes:

>> It would be nice if there was a built-in customization which caused
>> *Completions* to update as you type, as long as that buffer is visible.
>> I imagine such a request has been made before, so what is the obstacle
>> to adding it?
>
> I don't remember what was the obstacle, but here is the previous patch
> that implements the behavior of zsh and is based on icomplete-mode.

Nice! Although again this is more features than I want - I just want
*Completions* to automatically update after it opens.

>> I would like to figure out a solution which everyone is happy with, and
>> then I would be happy to implement it.  It seems to me that it's just a
>> matter of, after each keystroke, triggering (for minibuffer completion)
>> minibuffer-completion-help or (for buffer completion) some new function
>> to populate *Completions* with all-completions output.
>
> There is one difference between icomplete-mode and zcomplete-mode:
> the former uses completion-all-sorted-completions where the
> recently used items are at the top of the completion list.
> Whereas the latter uses the same alphabetical sorting as TAB
> shows in the *Completions* buffer.  Maybe a new function
> should allow any sorting order?

This is orthogonal to my original topic, which is just updating
*Completions* as I type, without changing how it gets populated.

That being said, yes this may be nice.  But minibuffer-completion-help
already does sort the completions using display-sort-function, just like
completion-all-sorted-completions, so what's causing the difference in
behavior?

Honestly the main place I find myself wanting different sorting of
completions is for buffer completion - I'd prefer buffers to be sorted
by most-recently-used.  Maybe we can just add such an option?

Re: Updating *Completions* as you type

[Juri Linkov]

>>> It would be nice if there was a built-in customization which caused
>>> *Completions* to update as you type, as long as that buffer is visible.
>>> I imagine such a request has been made before, so what is the obstacle
>>> to adding it?
>>
>> I don't remember what was the obstacle, but here is the previous patch
>> that implements the behavior of zsh and is based on icomplete-mode.
>
> Nice! Although again this is more features than I want - I just want
> *Completions* to automatically update after it opens.

Probably it's possible to pare it down to less code with less features
that could basically do the same after toggling a new option.

> That being said, yes this may be nice.  But minibuffer-completion-help
> already does sort the completions using display-sort-function, just like
> completion-all-sorted-completions, so what's causing the difference in
> behavior?

It seems this is implementable in minibuffer-completion-help
by copying this code from completion-all-sorted-completions:

              (setq all (minibuffer--sort-by-position
                         (minibuffer--sort-preprocess-history
                          (substring string 0 base-size))
                         all))

> Honestly the main place I find myself wanting different sorting of
> completions is for buffer completion - I'd prefer buffers to be sorted
> by most-recently-used.  Maybe we can just add such an option?

Such an option would be nice.  Maybe the right way to support it
is to add a new sort function with the code above.  Then the caller
could provide such a function in the completion metadata.

Re: Updating *Completions* as you type

[sbaugh]

[-- Attachment #1: Type: text/plain, Size: 816 bytes --]

Juri Linkov <juri@linkov.net> writes:

>>>> It would be nice if there was a built-in customization which caused
>>>> *Completions* to update as you type, as long as that buffer is visible.
>>>> I imagine such a request has been made before, so what is the obstacle
>>>> to adding it?
>>>
>>> I don't remember what was the obstacle, but here is the previous patch
>>> that implements the behavior of zsh and is based on icomplete-mode.
>>
>> Nice! Although again this is more features than I want - I just want
>> *Completions* to automatically update after it opens.
>
> Probably it's possible to pare it down to less code with less features
> that could basically do the same after toggling a new option.

Yes, agreed.  Taking inspiration from zcomplete, I wrote this patch to
provide just this feature, thoughts?


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Add-completions-auto-update.patch --]
[-- Type: text/x-patch, Size: 3402 bytes --]

From fef547ece1d5cd3eebbc2fb7f51e51f15cfc2b4a Mon Sep 17 00:00:00 2001
From: Spencer Baugh <sbaugh@catern.com>
Date: Sat, 14 Oct 2023 14:27:23 -0400
Subject: [PATCH] Add completions-auto-update

It can be useful for the *Completions* buffer to automatically update
as you type.  That way you can see immediately what the next
completion operation will do, even if you've changed text in the
buffer since triggering completion.

* lisp/minibuffer.el (completions-auto-update): Add.
(completions-no-auto-update-commands): Add.
(completions--post-command): Add.
(minibuffer-completion-help): Add completions--post-command to
post-command-hook.
---
 lisp/minibuffer.el | 38 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)

diff --git a/lisp/minibuffer.el b/lisp/minibuffer.el
index 3e30b68d5e9..9995da9e4b7 100644
--- a/lisp/minibuffer.el
+++ b/lisp/minibuffer.el
@@ -2376,6 +2376,40 @@ completions--fit-window-to-buffer
         (resize-temp-buffer-window win))
     (fit-window-to-buffer win completions-max-height)))
 
+(defcustom completions-auto-update t
+  "If non-nil, update the *Completions* buffer as you type.
+
+This only affects the *Completions* buffer if it is already
+displayed."
+  :type '(choice (const :tag "*Completions* doesn't change as you type" nil)
+                 (const :tag "*Completions* updates as you type" t))
+  :version "30.1")
+
+(defconst completions-no-auto-update-commands
+  '(previous-history-element
+    next-history-element
+    previous-line-or-history-element
+    next-line-or-history-element
+    completion-at-point
+    minibuffer-complete-and-exit
+    minibuffer-force-complete-and-exit
+    minibuffer-next-completion
+    minibuffer-previous-completion
+    minibuffer-choose-completion)
+  "Commands to skip updating *Completions*")
+
+(defun completions--post-command ()
+  "Update a displayed *Completions* buffer after a change"
+  (when completions-auto-update
+    (while-no-input
+      (let ((non-essential t))
+        (when (and (get-buffer-window "*Completions*" 0)
+                   (not (memq this-command completions-no-auto-update-commands)))
+          (redisplay)
+          (if completion-in-region-mode
+              (completion-help-at-point)
+            (minibuffer-completion-help)))))))
+
 (defun minibuffer-completion-help (&optional start end)
   "Display a list of possible completions of the current minibuffer contents."
   (interactive)
@@ -2398,6 +2432,7 @@ minibuffer-completion-help
           ;; If there are no completions, or if the current input is already
           ;; the sole completion, then hide (previous&stale) completions.
           (minibuffer-hide-completions)
+          (remove-hook 'post-command-hook #'completions--post-command t)
           (if completions
               (completion--message "Sole completion")
             (unless completion-fail-discreetly
@@ -2449,6 +2484,9 @@ minibuffer-completion-help
             (body-function
              . ,#'(lambda (_window)
                     (with-current-buffer mainbuf
+                      (when completions-auto-update
+                        (add-hook 'post-command-hook #'completions--post-command nil t))
+
                       ;; Remove the base-size tail because `sort' requires a properly
                       ;; nil-terminated list.
                       (when last (setcdr last nil))
-- 
2.41.0


[-- Attachment #3: Type: text/plain, Size: 2714 bytes --]


>> That being said, yes this may be nice.  But minibuffer-completion-help
>> already does sort the completions using display-sort-function, just like
>> completion-all-sorted-completions, so what's causing the difference in
>> behavior?
>
> It seems this is implementable in minibuffer-completion-help
> by copying this code from completion-all-sorted-completions:
>
>               (setq all (minibuffer--sort-by-position
>                          (minibuffer--sort-preprocess-history
>                           (substring string 0 base-size))
>                          all))

Oh, very interesting!  Maybe completions-sort should accept a new symbol
'history to behave this way?

I already think this would be very nice for project-switch-project, for
example, since I'm often switching between a few related projects.

Perhaps 'history should break ties by alphabetizing, otherwise
e.g. filenames you haven't visited before would be unsorted randomly.

Hmm, actually the case of filenames is a bit complex.  Because for
filenames, the strings we're completing over don't appear verbatim in
the history.  So perhaps read-file-name would need its own specialized
sorting function, which whenever you're completing in some directory,
sorts to the front any files in that directory whose full path appears
in file-name-history.

>> Honestly the main place I find myself wanting different sorting of
>> completions is for buffer completion - I'd prefer buffers to be sorted
>> by most-recently-used.  Maybe we can just add such an option?
>
> Such an option would be nice.  Maybe the right way to support it
> is to add a new sort function with the code above.  Then the caller
> could provide such a function in the completion metadata.

Oh, that's also interesting.  So there would be a function that the
completion metadata could specify as display-sort-function, which would
have the behavior of sorting based on history?

That also makes a lot of sense, and would allow commands like
project-switch-project and read-buffer to opt in on a command-by-command
basis, which might be more sensible.

So maybe adding 'history as a new option for completions-sort isn't a
good idea.  Instead we should just add user options for enabling history
sorting for files and buffers.  (And perhaps we could just enable
history sorting by default for project-switch-project.)

Also: it might be nice to switch between history-sorting and
alphabetized-sorting during the course of completion, both for files and
buffers.  Maybe we could do that by making a command which puts
completion into a mode where it just ignores the display-sort-function
specified by the completion metadata, and just always uses
completions-sort.

Re: Updating *Completions* as you type

[Eli Zaretskii]

> From: sbaugh@catern.com
> Date: Sat, 14 Oct 2023 16:05:11 -0400
> 
> Yes, agreed.  Taking inspiration from zcomplete, I wrote this patch to
> provide just this feature, thoughts?

I tried this.  The update is slow (most probably because it works off
the post-command-hook), and the UX is therefore extremely unpleasant
if you type fast enough.

> +(defcustom completions-auto-update t
> +  "If non-nil, update the *Completions* buffer as you type.

Thus _must_ be nil by default.

> +(defconst completions-no-auto-update-commands
> +  '(previous-history-element
> +    next-history-element
> +    previous-line-or-history-element
> +    next-line-or-history-element
> +    completion-at-point
> +    minibuffer-complete-and-exit
> +    minibuffer-force-complete-and-exit
> +    minibuffer-next-completion
> +    minibuffer-previous-completion
> +    minibuffer-choose-completion)
> +  "Commands to skip updating *Completions*")

Why are those excluded?  And why is this a defconst, not a defvar or
defcustom?

> +    (while-no-input
> +      (let ((non-essential t))
                ^^^^^^^^^^^^^^^
Why?

> +        (when (and (get-buffer-window "*Completions*" 0)
> +                   (not (memq this-command completions-no-auto-update-commands)))
> +          (redisplay)
             ^^^^^^^^^^^
Why do you need this?

Re: Updating *Completions* as you type

[sbaugh]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: sbaugh@catern.com
>> Date: Sat, 14 Oct 2023 16:05:11 -0400
>> 
>> Yes, agreed.  Taking inspiration from zcomplete, I wrote this patch to
>> provide just this feature, thoughts?
>
> I tried this.  The update is slow (most probably because it works off
> the post-command-hook), and the UX is therefore extremely unpleasant
> if you type fast enough.

Interesting.  Can you say more about what completion you're doing for
which the update is slow?  I would have expected the while-no-input to
cause the update to not block you if you type fast.

Of course, if there are optimizations that need to be done for
completion-generation, I'm happy to do them.

>> +(defcustom completions-auto-update t
>> +  "If non-nil, update the *Completions* buffer as you type.
>
> Thus _must_ be nil by default.

Of course, just defaulted to t to make testing the patch easier.

>> +(defconst completions-no-auto-update-commands
>> +  '(previous-history-element
>> +    next-history-element
>> +    previous-line-or-history-element
>> +    next-line-or-history-element
>> +    completion-at-point
>> +    minibuffer-complete-and-exit
>> +    minibuffer-force-complete-and-exit
>> +    minibuffer-next-completion
>> +    minibuffer-previous-completion
>> +    minibuffer-choose-completion)
>> +  "Commands to skip updating *Completions*")
>
> Why are those excluded?  And why is this a defconst, not a defvar or
> defcustom?

These are basically just commands where it would be annoying or
pointless to do an auto-update.

There are three classes here:

1. completion-at-point
   minibuffer-complete-and-exit
   minibuffer-force-complete-and-exit

   These commands themselves update *Completions*, so it's not necessary
   to update *Completions* immediately after they run, since it will
   already be up to date for the buffer text.

   This is just a performance thing and isn't strictly necessary.  I'll
   drop them from the next version.

   (The performance optimization could be implemented separately from
   completions-auto-update and apply even if it was nil.  We could have
   a general mechanism to skip regeneration of completions if the
   completion input hasn't changed.  That could be nice, although it's a
   complex topic.)
   
2. previous-history-element
   next-history-element
   previous-line-or-history-element
   next-line-or-history-element
   minibuffer-choose-completion

   These insert some text in the minibuffer which usually (but not
   always) will only have one completion.

   I'll drop these from the next version of my patch, since actually I
   now realize there's no reason to special-case skipping these.

3. minibuffer-next-completion
   minibuffer-previous-completion

   These call next-completion, which changes the location of point in
   *Completions*.  And if we auto-update *Completions*, we lose the
   location of point.

   But actually, even without completions-auto-update, it would be
   generally useful to not lose the location of point in *Completions*.
   So I'll just implement that as a separate patch.  And then I can also
   drop these two special cases.

So, in the next version of my patch, this variable will be dropped
entirely.

>> +    (while-no-input
>> +      (let ((non-essential t))
>                 ^^^^^^^^^^^^^^^
> Why?

This I borrowed from zcomplete.  It seems sensible, since
non-essential's docstring says:

  E.g., it can be used to prevent Tramp from prompting the user for a
  password when we are [...] displaying possible completions before the
  user even asked for it.

>> +        (when (and (get-buffer-window "*Completions*" 0)
>> +                   (not (memq this-command completions-no-auto-update-commands)))
>> +          (redisplay)
>              ^^^^^^^^^^^
> Why do you need this?

post-command-hook's docstring says:

  It is a bad idea to use this hook for expensive processing.  If
  unavoidable, wrap your code in ‘(while-no-input (redisplay) CODE)’ to
  avoid making Emacs unresponsive while the user types.

Re: Updating *Completions* as you type

[Eli Zaretskii]

> From: sbaugh@catern.com
> Date: Sun, 15 Oct 2023 11:55:47 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: sbaugh@catern.com
> >> Date: Sat, 14 Oct 2023 16:05:11 -0400
> >> 
> >> Yes, agreed.  Taking inspiration from zcomplete, I wrote this patch to
> >> provide just this feature, thoughts?
> >
> > I tried this.  The update is slow (most probably because it works off
> > the post-command-hook), and the UX is therefore extremely unpleasant
> > if you type fast enough.
> 
> Interesting.  Can you say more about what completion you're doing for
> which the update is slow?

I started by typing "C-x C-f TAB".

> I would have expected the while-no-input to cause the update to not
> block you if you type fast.

while-no-input lets me type, but does nothing to speed up the updates
of the *Completions* buffer; quite the contrary: it leaves it
outdated.

> >> +    (while-no-input
> >> +      (let ((non-essential t))
> >                 ^^^^^^^^^^^^^^^
> > Why?
> 
> This I borrowed from zcomplete.  It seems sensible, since
> non-essential's docstring says:
> 
>   E.g., it can be used to prevent Tramp from prompting the user for a
>   password when we are [...] displaying possible completions before the
>   user even asked for it.

So completion on remote files will not be able to benefit from this?

> >> +        (when (and (get-buffer-window "*Completions*" 0)
> >> +                   (not (memq this-command completions-no-auto-update-commands)))
> >> +          (redisplay)
> >              ^^^^^^^^^^^
> > Why do you need this?
> 
> post-command-hook's docstring says:
> 
>   It is a bad idea to use this hook for expensive processing.  If
>   unavoidable, wrap your code in ‘(while-no-input (redisplay) CODE)’ to
>   avoid making Emacs unresponsive while the user types.

Did you try this without the 'redisplay' call?

Re: Updating *Completions* as you type

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

Hi Eli,

>> >> +    (while-no-input
>> >> +      (let ((non-essential t))
>> >                 ^^^^^^^^^^^^^^^
>> > Why?
>>
>> This I borrowed from zcomplete.  It seems sensible, since
>> non-essential's docstring says:
>>
>>   E.g., it can be used to prevent Tramp from prompting the user for a
>>   password when we are [...] displaying possible completions before the
>>   user even asked for it.
>
> So completion on remote files will not be able to benefit from this?

No, remote files are not kicked out by this. non-essential just prevents
to open a *new* connection during the completion phase, when there
doesn't exist one already. If the connection exists already,
non-essential isn't relevant.

However, I believe it isn't needed here. Updating *Completions* will
happen only if a completion package is active. And they bind
non-essential already.

Best regards, Michael.

RE: [External] : Re: Updating *Completions* as you type

[Drew Adams]

> > So completion on remote files will not be able
> > to benefit from this?
> 
> No, remote files are not kicked out by this.
> non-essential just prevents to open a *new*
> connection during the completion phase

FWIW:

Icicles incremental completion is effectively turned
off whenever a remote file name is read, that is,
whenever your file-name input matches a remote-file
syntax.  (You can turn it back on using `C-#'.)

Option `icicle-test-for-remote-files-flag' controls this.
By default it's non-nil.

 Non-nil means Icicles tests for remote file names.
 A value of nil turns off all handling of remote file
 names by Tramp, including file-name completion.

 The testing due to a non-nil value takes a little time,
 but the test result saves time with Tramp handling, and
 it is used to avoid some other costly operations when a
 file is determined to be remote.  These operations are
 (a) incremental completion and (b) highlighting of the
 part of your current input that does not complete.

 Use a nil value only when you are sure that the file
 names you are completing are local.  The effect will be
 a slight speed increase for operations (a) and (b) for
 local files.  You can toggle this option from the
 minibuffer using `C-^'.

IOW, if you know that you won't be using remote file
names for a while, you can let Icicles and Tramp know
this by using `C-^' in the minibuffer to turn off the
option.

The "remote" test used is essentially `file-remote-p',
except that on MS Windows non-nil option
`icicle-network-drive-means-remote-flag' means consider
also files on a mapped network drive to be remote.

Re: Updating *Completions* as you type

[sbaugh]

[-- Attachment #1: Type: text/plain, Size: 354 bytes --]

sbaugh@catern.com writes:
>    But actually, even without completions-auto-update, it would be
>    generally useful to not lose the location of point in *Completions*.
>    So I'll just implement that as a separate patch.

Here's this; it's independently useful and independently installable.
Should be a nice, if a bit niche, improvement to behavior.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Keep-point-on-the-same-completion-in-the-completions.patch --]
[-- Type: text/x-patch, Size: 12752 bytes --]

From 64c08696755c29d2d9125734865ef93af6833fef Mon Sep 17 00:00:00 2001
From: Spencer Baugh <sbaugh@catern.com>
Date: Sun, 15 Oct 2023 16:47:16 -0400
Subject: [PATCH] Keep point on the same completion in the completions buffer

Currently if the user re-runs minibuffer-completion-help, point is
reset to the beginning of the buffer.  This throws away information
unnecessarily; let's keep point on the same completion the user
previously selected.

We move setting cursor-face-highlight-nonselected-window to
completion-setup-function so that the selected completion continues to
be highlighted after minibuffer-completion-help, which creates a new
*Completions* buffer.

* lisp/minibuffer.el (completion--insert-strings)
(completion--selected-posn, completion--insert-horizontal)
(completion--insert-vertical, completion--insert-one-column)
(completion--insert, display-completion-list): Add SELECTED argument.
(minibuffer-next-completion): Don't set
cursor-face-highlight-nonselected-window.
(minibuffer-completion-help): Calculate current-completion and pass it
to display-completion-list.
* lisp/simple.el (completions--get-posn): Add.
(choose-completion): Call completions--get-posn.
(completion-setup-function): Set
cursor-face-highlight-nonselected-window.
---
 lisp/minibuffer.el | 64 +++++++++++++++++++++++++++++++---------------
 lisp/simple.el     | 41 ++++++++++++++++-------------
 2 files changed, 66 insertions(+), 39 deletions(-)

diff --git a/lisp/minibuffer.el b/lisp/minibuffer.el
index 2120e31775e..998ef9f05a9 100644
--- a/lisp/minibuffer.el
+++ b/lisp/minibuffer.el
@@ -2034,12 +2034,17 @@ completions-header-format
                  (string :tag "Format string for heading line"))
   :version "29.1")
 
-(defun completion--insert-strings (strings &optional group-fun)
+(defvar completion--selected-posn)
+
+(defun completion--insert-strings (strings &optional group-fun selected)
   "Insert a list of STRINGS into the current buffer.
 The candidate strings are inserted into the buffer depending on the
 completions format as specified by the variable `completions-format'.
 Runs of equal candidate strings are eliminated.  GROUP-FUN is a
-`group-function' used for grouping the completion candidates."
+`group-function' used for grouping the completion candidates.
+
+If SELECTED exists in STRINGS, point is set to its first
+instance; otherwise, it's set to `point-min'."
   (when (consp strings)
     (let* ((length (apply #'max
 			  (mapcar (lambda (s)
@@ -2055,18 +2060,20 @@ completion--insert-strings
 		     ;; Don't allocate more columns than we can fill.
 		     ;; Windows can't show less than 3 lines anyway.
 		     (max 1 (/ (length strings) 2))))
-	   (colwidth (/ wwidth columns)))
+	   (colwidth (/ wwidth columns))
+           completion--selected-posn)
       (unless (or tab-stop-list (null completion-tab-width)
                   (zerop (mod colwidth completion-tab-width)))
         ;; Align to tab positions for the case
         ;; when the caller uses tabs inside prefix.
         (setq colwidth (- colwidth (mod colwidth completion-tab-width))))
       (funcall (intern (format "completion--insert-%s" completions-format))
-               strings group-fun length wwidth colwidth columns))))
+               strings group-fun length wwidth colwidth columns selected)
+      (goto-char (or completion--selected-posn (point-min))))))
 
 (defun completion--insert-horizontal (strings group-fun
                                               length wwidth
-                                              colwidth _columns)
+                                              colwidth _columns selected)
   (let ((column 0)
         (first t)
 	(last-title nil)
@@ -2103,7 +2110,7 @@ completion--insert-horizontal
 				 `(display (space :align-to ,column)))
 	    nil))
         (setq first nil)
-        (completion--insert str group-fun)
+        (completion--insert str group-fun selected)
 	;; Next column to align to.
 	(setq column (+ column
 			;; Round up to a whole number of columns.
@@ -2111,7 +2118,7 @@ completion--insert-horizontal
 
 (defun completion--insert-vertical (strings group-fun
                                             _length _wwidth
-                                            colwidth columns)
+                                            colwidth columns selected)
   (while strings
     (let ((group nil)
           (column 0)
@@ -2155,13 +2162,15 @@ completion--insert-vertical
 	    (insert " \t")
 	    (set-text-properties (1- (point)) (point)
 			         `(display (space :align-to ,column))))
-          (completion--insert str group-fun)
+          (completion--insert str group-fun selected)
 	  (if (> column 0)
 	      (forward-line)
 	    (insert "\n"))
 	  (setq row (1+ row)))))))
 
-(defun completion--insert-one-column (strings group-fun &rest _)
+(defun completion--insert-one-column (strings group-fun
+                                              _length _wwidth
+                                              _colwidth _columns selected)
   (let ((last-title nil) (last-string nil))
     (dolist (str strings)
       (unless (equal last-string str) ; Remove (consecutive) duplicates.
@@ -2172,11 +2181,14 @@ completion--insert-one-column
               (setq last-title title)
               (when title
                 (insert (format completions-group-format title) "\n")))))
-        (completion--insert str group-fun)
+        (completion--insert str group-fun selected)
         (insert "\n")))
     (delete-char -1)))
 
-(defun completion--insert (str group-fun)
+(defun completion--insert (str group-fun selected)
+  (when (and (not completion--selected-posn)
+             (equal (or (car-safe str) str) selected))
+    (setq completion--selected-posn (point)))
   (if (not (consp str))
       (add-text-properties
        (point)
@@ -2197,7 +2209,7 @@ completion--insert
         (let ((beg (point))
               (end (progn (insert prefix) (point))))
           (add-text-properties beg end `(mouse-face nil completion--string ,(car str)))))
-      (completion--insert (car str) group-fun)
+      (completion--insert (car str) group-fun selected)
       (let ((beg (point))
             (end (progn (insert suffix) (point))))
         (add-text-properties beg end `(mouse-face nil completion--string ,(car str)))
@@ -2267,7 +2279,7 @@ completion-hilit-commonality
         completions)
        base-size))))
 
-(defun display-completion-list (completions &optional common-substring group-fun)
+(defun display-completion-list (completions &optional common-substring group-fun selected)
   "Display the list of completions, COMPLETIONS, using `standard-output'.
 Each element may be just a symbol or string
 or may be a list of two strings to be printed as if concatenated.
@@ -2276,6 +2288,8 @@ display-completion-list
 `standard-output' must be a buffer.
 The actual completion alternatives, as inserted, are given `mouse-face'
 properties of `highlight'.
+If SELECTED exists in COMPLETIONS, point is set to its first
+instance; otherwise, it's set to `point-min'.
 At the end, this runs the normal hook `completion-setup-hook'.
 It can find the completion buffer in `standard-output'.
 GROUP-FUN is a `group-function' used for grouping the completion
@@ -2299,9 +2313,15 @@ display-completion-list
       (goto-char (point-max))
       (when completions-header-format
         (insert (format completions-header-format (length completions))))
-      (completion--insert-strings completions group-fun)))
-
-  (run-hooks 'completion-setup-hook)
+      (completion--insert-strings completions group-fun selected)))
+
+  ;; Make sure point stays at SELECTED.
+  (let ((marker
+         (when (bufferp standard-output)
+           (with-current-buffer standard-output (point-marker)))))
+    (run-hooks 'completion-setup-hook)
+    (when marker
+      (with-current-buffer standard-output (goto-char marker))))
   nil)
 
 (defvar completion-extra-properties nil
@@ -2421,7 +2441,10 @@ minibuffer-completion-help
              ;; window, mark it as softly-dedicated, so bury-buffer in
              ;; minibuffer-hide-completions will know whether to
              ;; delete the window or not.
-             (display-buffer-mark-dedicated 'soft))
+             (display-buffer-mark-dedicated 'soft)
+             (current-completion
+              (when-let ((buf (get-buffer "*Completions*")))
+                (with-current-buffer buf (completions--get-posn (point))))))
         (with-current-buffer-window
           "*Completions*"
           ;; This is a copy of `display-buffer-fallback-action'
@@ -2440,7 +2463,7 @@ minibuffer-completion-help
             ,(when temp-buffer-resize-mode
                '(preserve-size . (nil . t)))
             (body-function
-             . ,#'(lambda (_window)
+             . ,#'(lambda (window)
                     (with-current-buffer mainbuf
                       ;; Remove the base-size tail because `sort' requires a properly
                       ;; nil-terminated list.
@@ -2527,7 +2550,8 @@ minibuffer-completion-help
                                                      (if (eq (car bounds) (length result))
                                                          'exact 'finished)))))))
 
-                      (display-completion-list completions nil group-fun)))))
+                      (display-completion-list completions nil group-fun current-completion)
+                      (set-window-point window (with-current-buffer standard-output (point)))))))
           nil)))
     nil))
 
@@ -4496,8 +4520,6 @@ minibuffer-next-completion
   (interactive "p")
   (let ((auto-choose minibuffer-completion-auto-choose))
     (with-minibuffer-completions-window
-      (when completions-highlight-face
-        (setq-local cursor-face-highlight-nonselected-window t))
       (next-completion (or n 1))
       (when auto-choose
         (let ((completion-use-base-affixes t))
diff --git a/lisp/simple.el b/lisp/simple.el
index ec14bec9e07..3ab8b783659 100644
--- a/lisp/simple.el
+++ b/lisp/simple.el
@@ -10000,6 +10000,25 @@ next-completion
     (when (/= 0 n)
       (switch-to-minibuffer))))
 
+(defun completions--get-posn (position)
+  "Return the completion at POSITION as a string."
+  (save-excursion
+    (goto-char position)
+    (let (beg)
+      (cond
+       ((and (not (eobp))
+             (get-text-property (point) 'completion--string))
+        (setq beg (1+ (point))))
+       ((and (not (bobp))
+             (get-text-property (1- (point)) 'completion--string))
+        (setq beg (point))))
+      (when beg
+        (setq beg (or (previous-single-property-change
+                       beg 'completion--string)
+                      beg))
+        (substring-no-properties
+         (get-text-property beg 'completion--string))))))
+
 (defun choose-completion (&optional event no-exit no-quit)
   "Choose the completion at point.
 If EVENT, use EVENT's position to determine the starting position.
@@ -10019,24 +10038,8 @@ choose-completion
           (base-affixes completion-base-affixes)
           (insert-function completion-list-insert-choice-function)
           (completion-no-auto-exit (if no-exit t completion-no-auto-exit))
-          (choice
-           (save-excursion
-             (goto-char (posn-point (event-start event)))
-             (let (beg)
-               (cond
-                ((and (not (eobp))
-                      (get-text-property (point) 'completion--string))
-                 (setq beg (1+ (point))))
-                ((and (not (bobp))
-                      (get-text-property (1- (point)) 'completion--string))
-                 (setq beg (point)))
-                (t (error "No completion here")))
-               (setq beg (or (previous-single-property-change
-                              beg 'completion--string)
-                             beg))
-               (substring-no-properties
-                (get-text-property beg 'completion--string))))))
-
+          (choice (or (completions--get-posn (posn-point (event-start event)))
+                      (error "No completion here"))))
       (unless (buffer-live-p buffer)
         (error "Destination buffer is dead"))
       (unless no-quit
@@ -10208,6 +10211,8 @@ completion-setup-function
             (base-affixes completion-base-affixes)
             (insert-fun completion-list-insert-choice-function))
         (completion-list-mode)
+        (when completions-highlight-face
+          (setq-local cursor-face-highlight-nonselected-window t))
         (setq-local completion-base-position base-position)
         (setq-local completion-base-affixes base-affixes)
         (setq-local completion-list-insert-choice-function insert-fun))
-- 
2.41.0

Re: Updating *Completions* as you type

[Juri Linkov]

> Currently if the user re-runs minibuffer-completion-help, point is
> reset to the beginning of the buffer.  This throws away information
> unnecessarily; let's keep point on the same completion the user
> previously selected.

Are you sure everyone would want to always preselect the previous
candidate in the middle of completions in every new completion?

Re: Updating *Completions* as you type

[Spencer Baugh]

Juri Linkov <juri@linkov.net> writes:
>> Currently if the user re-runs minibuffer-completion-help, point is
>> reset to the beginning of the buffer.  This throws away information
>> unnecessarily; let's keep point on the same completion the user
>> previously selected.
>
> Are you sure everyone would want to always preselect the previous
> candidate in the middle of completions in every new completion?

Definitely not.  I allowed that behavior just to demonstrate the patch.
The real version will only retrieve the previous completion when
Completions is currently displayed - in other words, it will forget
whenever Completions disappears.

Re: Updating *Completions* as you type

[Juri Linkov]

>>> Nice! Although again this is more features than I want - I just want
>>> *Completions* to automatically update after it opens.
>>
>> Probably it's possible to pare it down to less code with less features
>> that could basically do the same after toggling a new option.
>
> Yes, agreed.  Taking inspiration from zcomplete, I wrote this patch to
> provide just this feature, thoughts?

Thank you very much for finally bringing this useful feature to the core
while keeping it sufficiently short.  I briefly tried your patch, and
it works exactly as expected by experience of using such common completions
as in the address bar of web browsers, etc.  I'll continue testing it more
for a while.

Re: Updating *Completions* as you type

[Rudolf Adamkovič]

Juri Linkov <juri@linkov.net> writes:

> Thank you very much for finally bringing this useful feature to the core
> while keeping it sufficiently short.

I, too, would like to see both automatic updates and MRU sorting in the
default minibuffer completion UI.  I hacked together both for myself,
only to return to Vertico for its incredible performance.  So, I think
that automatic updates must be fast, ideally as fast as Vertico.

It is also worth noting that the recent "restart" of MCT development
indicates that there is wider interest in this functionality:

https://git.sr.ht/~protesilaos/mct/commit/2cbf74edb4f4553d7075b34e06adcf59e96efda2

Rudy
-- 
"Be especially critical of any statement following the word
'obviously.'"
-- Anna Pell Wheeler, 1883-1966

Rudolf Adamkovič <salutis@me.com> [he/him]
Studenohorská 25
84103 Bratislava
Slovakia

Re: Updating *Completions* as you type

[Juri Linkov]

> I, too, would like to see both automatic updates and MRU sorting in the
> default minibuffer completion UI.  I hacked together both for myself,
> only to return to Vertico for its incredible performance.  So, I think
> that automatic updates must be fast, ideally as fast as Vertico.
>
> It is also worth noting that the recent "restart" of MCT development
> indicates that there is wider interest in this functionality:
>
> https://git.sr.ht/~protesilaos/mct/commit/2cbf74edb4f4553d7075b34e06adcf59e96efda2

Thanks for the link, cross-pollination of ideas would benefit everyone.

Re: Updating *Completions* as you type

[Eshel Yaron]

Hi,

sbaugh@catern.com writes:

> Taking inspiration from zcomplete, I wrote this patch to
> provide just this feature, thoughts?
>
> From fef547ece1d5cd3eebbc2fb7f51e51f15cfc2b4a Mon Sep 17 00:00:00 2001
> From: Spencer Baugh <sbaugh@catern.com>
> Date: Sat, 14 Oct 2023 14:27:23 -0400
> Subject: [PATCH] Add completions-auto-update
>
> It can be useful for the *Completions* buffer to automatically update
> as you type.  That way you can see immediately what the next
> completion operation will do, even if you've changed text in the
> buffer since triggering completion.
>
> * lisp/minibuffer.el (completions-auto-update): Add.
> (completions-no-auto-update-commands): Add.
> (completions--post-command): Add.
> (minibuffer-completion-help): Add completions--post-command to
> post-command-hook.

I tried this out, and I came across a certain issue: say you do
`C-h o foo TAB`, and then looking at the completion candidates
you realize you actually wanted something with `bar`, so you
type `M-DEL` to delete `foo` from the minibuffer to make room
for `bar`.  But now `post-command-hook` runs with an empty
minibuffer before you start typing `bar`, so it updates the
completions buffer with a very large list of symbols, causing
Emacs to briefly stall.  I think that the completions buffer
possibly shouldn't be updated when the minibuffer is empty.
Perhaps there could be a minimum minibuffer contents size for
updating the completions buffer, or some other heuristic to
avoid auto-displaying excessively large sets of candidates.
Another option is to update the completions buffer on
`post-self-insert-hook` instead of `post-command-hook`, that
way you only update the completions buffer when the user
actually inserts input in the minibuffer, and you don't need to
keep a `completions-no-auto-update-commands` list.


Best,

Eshel

RE: [External] : Re: Updating *Completions* as you type

[Drew Adams]

> I think that the completions buffer possibly
> shouldn't be updated when the minibuffer is empty.
> Perhaps there could be a minimum minibuffer
> contents size for updating the completions buffer,
> or some other heuristic to avoid auto-displaying
> excessively large sets of candidates.

FWIW:

In addition to the Icicles variables I mentioned
previously, there's this one:

4. `icicle-Completions-display-min-chars':

  `*Completions*' window is removed if fewer chars
  than this are input (i.e., in the minibuffer).

Re: Updating *Completions* as you type

[Juri Linkov]

> I tried this out, and I came across a certain issue: say you do
> `C-h o foo TAB`, and then looking at the completion candidates
> you realize you actually wanted something with `bar`, so you
> type `M-DEL` to delete `foo` from the minibuffer to make room
> for `bar`.  But now `post-command-hook` runs with an empty
> minibuffer before you start typing `bar`, so it updates the
> completions buffer with a very large list of symbols, causing
> Emacs to briefly stall.  I think that the completions buffer
> possibly shouldn't be updated when the minibuffer is empty.

This looks like icomplete-show-matches-on-no-input.

> Perhaps there could be a minimum minibuffer contents size for
> updating the completions buffer, or some other heuristic to
> avoid auto-displaying excessively large sets of candidates.

And this is like icomplete-max-delay-chars.

> Another option is to update the completions buffer on
> `post-self-insert-hook` instead of `post-command-hook`, that
> way you only update the completions buffer when the user
> actually inserts input in the minibuffer, and you don't need to
> keep a `completions-no-auto-update-commands` list.

Is `post-self-insert-hook` called for pasting a string?
Maybe better to replace blocklist with allowlist that contains
a list '(self-insert-command yank)' by default as discussed in
https://yhetil.org/emacs-devel/87eefp18q2.fsf@gnu.org/

Re: Updating *Completions* as you type

[sbaugh]

[-- Attachment #1: Type: text/plain, Size: 3751 bytes --]

sbaugh@catern.com writes:
>>> That being said, yes this may be nice.  But minibuffer-completion-help
>>> already does sort the completions using display-sort-function, just like
>>> completion-all-sorted-completions, so what's causing the difference in
>>> behavior?
>>
>> It seems this is implementable in minibuffer-completion-help
>> by copying this code from completion-all-sorted-completions:
>>
>>               (setq all (minibuffer--sort-by-position
>>                          (minibuffer--sort-preprocess-history
>>                           (substring string 0 base-size))
>>                          all))
>
>Oh, very interesting!  Maybe completions-sort should accept a new symbol
>'history to behave this way?
>
>I already think this would be very nice for project-switch-project, for
>example, since I'm often switching between a few related projects.
>
>Perhaps 'history should break ties by alphabetizing, otherwise
>e.g. filenames you haven't visited before would be unsorted randomly.
>
>Hmm, actually the case of filenames is a bit complex.  Because for
>filenames, the strings we're completing over don't appear verbatim in
>the history.  So perhaps read-file-name would need its own specialized
>sorting function, which whenever you're completing in some directory,
>sorts to the front any files in that directory whose full path appears
>in file-name-history.
>
>>> Honestly the main place I find myself wanting different sorting of
>>> completions is for buffer completion - I'd prefer buffers to be sorted
>>> by most-recently-used.  Maybe we can just add such an option?
>>
>> Such an option would be nice.  Maybe the right way to support it
>> is to add a new sort function with the code above.  Then the caller
>> could provide such a function in the completion metadata.
>
>Oh, that's also interesting.  So there would be a function that the
>completion metadata could specify as display-sort-function, which would
>have the behavior of sorting based on history?
>
>That also makes a lot of sense, and would allow commands like
>project-switch-project and read-buffer to opt in on a command-by-command
>basis, which might be more sensible.
>
>So maybe adding 'history as a new option for completions-sort isn't a
>good idea.  Instead we should just add user options for enabling history
>sorting for files and buffers.  (And perhaps we could just enable
>history sorting by default for project-switch-project.)
>
>Also: it might be nice to switch between history-sorting and
>alphabetized-sorting during the course of completion, both for files and
>buffers.  Maybe we could do that by making a command which puts
>completion into a mode where it just ignores the display-sort-function
>specified by the completion metadata, and just always uses
>completions-sort.

On reflection, I think adding a new option for completions-sort is
indeed what we should do.  Otherwise, we won't get historical sorting
for non-programmed completing-read, which is a pretty common kind of
completing-read.  Here's a patch to do that.

We can always add display-sort-functions later for buffers and files, if
that ends up being desirable.

Also, tangentially, I think probably we should rework
minibuffer-complete-history and minibuffer-complete-defaults to be
persistent - as in, regular TAB afterwards continues to complete history
or defaults.  And there should be some way to reset back to normal.
That would be a good complement to this completions-sort change, by
maybe giving a way to switch on-demand to alphabetical sorting.  (I've
long thought this would be good and useful, but in particular it's
relevant for completions-auto-update since that will otherwise nearly
immediately reset the displayed completions back to normal.)


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Add-a-historical-option-to-completions-sort.patch --]
[-- Type: text/x-patch, Size: 3851 bytes --]

From b97e311ee42f7f9021b3c0e017636e258dd6d5d9 Mon Sep 17 00:00:00 2001
From: Spencer Baugh <sbaugh@catern.com>
Date: Tue, 17 Oct 2023 09:09:55 -0400
Subject: [PATCH] Add a historical option to completions-sort

This causes completion to sort based on history.  This is useful for
getting, e.g., most-recently-used order from C-x b.

We only do historical sorting when there's a completion-specific
history variable.  Otherwise, candidate would be spuriously brought to
the front if they just happened to match a string that the user has
entered before.

Also, the presence of a completion-specific history variable is a
decent proxy for "does history matter for this command?"; if there
isn't a specific history variable, sorting based on history would
probably be undesirable.  (Note also that the user can always choose
to do historical sorting mid-completion by running
minibuffer-complete-history.)

* lisp/minibuffer.el (completions-sort): Document 'historical option.
(minibuffer-completion-help): Support 'historical option.
---
 lisp/minibuffer.el | 31 ++++++++++++++++++++++++++-----
 1 file changed, 26 insertions(+), 5 deletions(-)

diff --git a/lisp/minibuffer.el b/lisp/minibuffer.el
index f53cd739e2f..782efff80f9 100644
--- a/lisp/minibuffer.el
+++ b/lisp/minibuffer.el
@@ -1313,14 +1313,26 @@ completion-cycle-threshold
 (defcustom completions-sort 'alphabetical
   "Sort candidates in the *Completions* buffer.
 
-The value can be nil to disable sorting, `alphabetical' for
-alphabetical sorting or a custom sorting function.  The sorting
-function takes and returns a list of completion candidate
-strings."
+Candidate completions in the *Completions* are sorted depending
+on the value.
+
+If nil, sorting is disabled.
+If `alphabetical', candidates are sorted alphabetically.
+If `historical', candidates are first sorted alphabetically, then
+candidates occurring in `minibuffer-history-variable' are moved
+to the front based on the order they occur in the history.
+If a function, the function is called to sort the candidates.
+The sorting function takes and returns a list of completion
+candidate strings.
+
+If the completion-specific metadata provides a
+`display-sort-function', that is used instead and this value is
+ignored."
   :type '(choice (const :tag "No sorting" nil)
                  (const :tag "Alphabetical sorting" alphabetical)
+                 (const :tag "Historical sorting" historical)
                  (function :tag "Custom function"))
-  :version "29.1")
+  :version "30.1")
 
 (defcustom completions-group nil
   "Enable grouping of completion candidates in the *Completions* buffer.
@@ -2510,6 +2522,15 @@ minibuffer-completion-help
                                           (pcase completions-sort
                                             ('nil completions)
                                             ('alphabetical (sort completions #'string-lessp))
+                                            ('historical
+                                             (let ((alphabetized (sort completions #'string-lessp)))
+                                               ;; Only use history when it's specific to these completions.
+                                               (if (eq minibuffer-history-variable 'minibuffer-history)
+                                                   alphabetized
+                                                 (minibuffer--sort-by-position
+                                                  (minibuffer--sort-preprocess-history
+                                                   (substring string 0 base-size))
+                                                  alphabetized))))
                                             (_ (funcall completions-sort completions)))))
 
                       ;; After sorting, group the candidates using the
-- 
2.41.0

Re: Updating *Completions* as you type

[Juri Linkov]

> On reflection, I think adding a new option for completions-sort is
> indeed what we should do.  Otherwise, we won't get historical sorting
> for non-programmed completing-read, which is a pretty common kind of
> completing-read.  Here's a patch to do that.

> We can always add display-sort-functions later for buffers and files, if
> that ends up being desirable.

Thanks, this is a nice first step that paves the way to later
additions of display-sort-functions and keys for toggling
sorting in *Completions*.

> Also, tangentially, I think probably we should rework
> minibuffer-complete-history and minibuffer-complete-defaults to be
> persistent - as in, regular TAB afterwards continues to complete history
> or defaults.  And there should be some way to reset back to normal.
> That would be a good complement to this completions-sort change, by
> maybe giving a way to switch on-demand to alphabetical sorting.  (I've
> long thought this would be good and useful, but in particular it's
> relevant for completions-auto-update since that will otherwise nearly
> immediately reset the displayed completions back to normal.)

I think we should choose a key to toggle completion type between
history/default/regular completion.

> Subject: [PATCH] Add a historical option to completions-sort
>
> This causes completion to sort based on history.  This is useful for
> getting, e.g., most-recently-used order from C-x b.

Actually for 'C-x b' I'd prefer to sort buffers by the order of (buffer-list),
not by the order buffers occur in the minibuffer history (that I'd like
to use for everything else).

Re: Updating *Completions* as you type

[Spencer Baugh]

Juri Linkov <juri@linkov.net> writes:
>> On reflection, I think adding a new option for completions-sort is
>> indeed what we should do.  Otherwise, we won't get historical sorting
>> for non-programmed completing-read, which is a pretty common kind of
>> completing-read.  Here's a patch to do that.
>
>> We can always add display-sort-functions later for buffers and files, if
>> that ends up being desirable.
>
> Thanks, this is a nice first step that paves the way to later
> additions of display-sort-functions and keys for toggling
> sorting in *Completions*.
>
>> Also, tangentially, I think probably we should rework
>> minibuffer-complete-history and minibuffer-complete-defaults to be
>> persistent - as in, regular TAB afterwards continues to complete history
>> or defaults.  And there should be some way to reset back to normal.
>> That would be a good complement to this completions-sort change, by
>> maybe giving a way to switch on-demand to alphabetical sorting.  (I've
>> long thought this would be good and useful, but in particular it's
>> relevant for completions-auto-update since that will otherwise nearly
>> immediately reset the displayed completions back to normal.)
>
> I think we should choose a key to toggle completion type between
> history/default/regular completion.

That works too of course, although it causes some more proliferation of
keys.

I'm curious, what is the intended usage of
minibuffer-complete-{history,defaults}?  The fact that they only do a
single completion has made them not very usable for me.

>> Subject: [PATCH] Add a historical option to completions-sort
>>
>> This causes completion to sort based on history.  This is useful for
>> getting, e.g., most-recently-used order from C-x b.
>
> Actually for 'C-x b' I'd prefer to sort buffers by the order of (buffer-list),
> not by the order buffers occur in the minibuffer history (that I'd like
> to use for everything else).

Reasonable.  I suggest that this should be achieved by adding a
display-sort-function, though.  (And... actually, that
display-sort-function could maybe just be identity, since the
completions are generated from buffer-list so they are in that order
anyway?)

RE: [External] : Re: Updating *Completions* as you type

[Drew Adams]

> > I think we should choose a key to toggle completion 
> > type between history/default/regular completion.
> 
> That works too of course, although it causes some 
> more proliferation of keys.

The approach I took in Icicles is preferable, IMO.
Food for thought:

1. There can be any number of sort orders to choose
   from at any given time, i.e., in any given context.
   For different calls to `completing-read' (e.g.
   for different commands), some very different sort
   orders can make sense.

   And one of the available ways of sorting is to do
   nothing - NOT sort the available candidates.  This
   can be useful if the completion function already
   provides them in a useful order, or if any sorting
   at all would be expensive for some reason.

2. Instead of wasting multiple keys, one for each
   sort-order choice, a single key cycles among them:
   `C-,'.

3. For any `completing-read' call there can also be
   a set of alternative sort orders.  You can toggle
   to the current alternative sort using `C-M-,'.
   And you can cycle among the available alternative
   orders using `M-,' (just like `C-,').

4. Both `C-,' and `M-,' also provide another way to
   choose a sort order, besides cycling to it: use
   completion to choose it by name.

5. You can choose, with an option, whether, by default,
   to (1) always use cycling to choose a sort order,
   (2) always use completion, or (3) cycle if there are
   fewer than N orders to choose from and complete if
   there are N or more.

6. Reversing the current sort order is just `C-,' with
   a numeric prefix arg, e.g., `C-9 C-,'.

7. Regardless of your user-option preference (#5), you
   can flip to the other kind of choosing as a one-off,
   on the fly, using a non-numeric prefix arg.  E.g.,
   if the option says to use cycling in the current
   situation then `C-u C-,' switches to choosing the
   order by name (completing).

8. It's very easy for users to define new sort orders.
   This is quite important, IMO.  It should seem clear,
   from your discussion so far, why this makes sense.

9. The available sort orders for `C-,' are defined by
   a user option.  But commands can adjust this list
   of possibilities by adding command-specific sort
   orders or removing some that might be inappropriate.

10. The sort order you last chose remains in effect
   for subsequent completion (e.g. other commands).
   But if the sort order isn't appropriate for some
   command it can turn sorting off, in which case
   `C-,' lets you choose an appropriate order. 

11. IMO, there is no reason at all to have different
   sort orders for display and for cycling:
   `display-sort-function' and `cycle-sort-function'.
   At least I haven't seen (or heard) any reason.
   Can anyone say why these aren't the same thing in
   Emacs?

   And even if there were some use case where someone
   might want them to differ, why wouldn't the normal,
   default case be for them to be the same (synced)?

   You're apparently already beginning to wonder about
   a difference between a "current" completion in
   *Completions* (the one that's highlighted) and a
   "current" completion in the minibuffer, no?  Why
   let them differ, ever - what's the use case?

   In my code (both Icicles and tiny `sortie.el') the
   two are the same.  The current candidate in the
   minibuffer is always the candidate highlighted in
   *Completions* (and vice versa - identical).

12. The current sort order, and whether it's reversed,
   is indicated in the mode-line of *Completions*, so
   you always know what sauce you're being cooked in.

https://www.emacswiki.org/emacs/Icicles_-_Completions_Display#SortOrder
   
13. I wonder if any of you have even looked at the doc
   describing the features I'm talking about.  I get
   the impression you're maybe starting to re-invent
   the wheel, without looking at how existing round
   things have been rolling along so far.

https://www.emacswiki.org/emacs/Icicles_-_Sorting_Candidates

(Many of the same sorting features are available with
`sortie.el' as with Icicles.  Trying it with library
`keysee.el' is a simple way to see what it's like.)

https://www.emacswiki.org/emacs/Sortie

https://www.emacswiki.org/emacs/KeySee

Re: Updating *Completions* as you type

[Juri Linkov]

>>> Also, tangentially, I think probably we should rework
>>> minibuffer-complete-history and minibuffer-complete-defaults to be
>>> persistent - as in, regular TAB afterwards continues to complete history
>>> or defaults.  And there should be some way to reset back to normal.
>>> That would be a good complement to this completions-sort change, by
>>> maybe giving a way to switch on-demand to alphabetical sorting.  (I've
>>> long thought this would be good and useful, but in particular it's
>>> relevant for completions-auto-update since that will otherwise nearly
>>> immediately reset the displayed completions back to normal.)
>>
>> I think we should choose a key to toggle completion type between
>> history/default/regular completion.
>
> That works too of course, although it causes some more proliferation of
> keys.
>
> I'm curious, what is the intended usage of
> minibuffer-complete-{history,defaults}?  The fact that they only do a
> single completion has made them not very usable for me.

The currently limited use case is that you can type a substring,
then TAB and select the completion from the list.  You are welcome
to improve this as well.

>> Actually for 'C-x b' I'd prefer to sort buffers by the order of (buffer-list),
>> not by the order buffers occur in the minibuffer history (that I'd like
>> to use for everything else).
>
> Reasonable.  I suggest that this should be achieved by adding a
> display-sort-function, though.  (And... actually, that
> display-sort-function could maybe just be identity, since the
> completions are generated from buffer-list so they are in that order
> anyway?)

Actually there is already a nil value in completions-sort with the tag
"No sorting".  This works nicely for 'C-x b'.  The remaining need is
to be able to set it only for 'C-x b', not for other completion types.

Re: Updating *Completions* as you type

[Spencer Baugh]

Juri Linkov <juri@linkov.net> writes:
>>>> Also, tangentially, I think probably we should rework
>>>> minibuffer-complete-history and minibuffer-complete-defaults to be
>>>> persistent - as in, regular TAB afterwards continues to complete history
>>>> or defaults.  And there should be some way to reset back to normal.
>>>> That would be a good complement to this completions-sort change, by
>>>> maybe giving a way to switch on-demand to alphabetical sorting.  (I've
>>>> long thought this would be good and useful, but in particular it's
>>>> relevant for completions-auto-update since that will otherwise nearly
>>>> immediately reset the displayed completions back to normal.)
>>>
>>> I think we should choose a key to toggle completion type between
>>> history/default/regular completion.
>>
>> That works too of course, although it causes some more proliferation of
>> keys.
>>
>> I'm curious, what is the intended usage of
>> minibuffer-complete-{history,defaults}?  The fact that they only do a
>> single completion has made them not very usable for me.
>
> The currently limited use case is that you can type a substring,
> then TAB and select the completion from the list.  You are welcome
> to improve this as well.
>
>>> Actually for 'C-x b' I'd prefer to sort buffers by the order of (buffer-list),
>>> not by the order buffers occur in the minibuffer history (that I'd like
>>> to use for everything else).
>>
>> Reasonable.  I suggest that this should be achieved by adding a
>> display-sort-function, though.  (And... actually, that
>> display-sort-function could maybe just be identity, since the
>> completions are generated from buffer-list so they are in that order
>> anyway?)
>
> Actually there is already a nil value in completions-sort with the tag
> "No sorting".  This works nicely for 'C-x b'.  The remaining need is
> to be able to set it only for 'C-x b', not for other completion types.

I feel quite strongly that we should not extend completions-sort to be
able to affect different completion types differently.  Instead I think
we should leave completions-sort as a blanket configuration for what to
do if the completion metadata does not specify display-sort-function,
and if we want to allow customizing an individual completion type, that
completion type should specify a display-sort-function which can be
customized.

If we do extend completions-sort to affect different completion types
differently, I expect we'll see lots of silly things, like packages with
new completion types which automatically install changes to
completions-sort instead of just specifying their own
display-sort-function.

Re: Updating *Completions* as you type

[Juri Linkov]

>> Actually there is already a nil value in completions-sort with the tag
>> "No sorting".  This works nicely for 'C-x b'.  The remaining need is
>> to be able to set it only for 'C-x b', not for other completion types.
>
> I feel quite strongly that we should not extend completions-sort to be
> able to affect different completion types differently.  Instead I think
> we should leave completions-sort as a blanket configuration for what to
> do if the completion metadata does not specify display-sort-function,
> and if we want to allow customizing an individual completion type, that
> completion type should specify a display-sort-function which can be
> customized.
>
> If we do extend completions-sort to affect different completion types
> differently, I expect we'll see lots of silly things, like packages with
> new completion types which automatically install changes to
> completions-sort instead of just specifying their own
> display-sort-function.

Probably different completion types should provide separate options
such as e.g. 'read-char-by-name-sort' defines 'display-sort-function'
for 'read-char-by-name'.

The same could be added for 'C-x b'.  But unfortunately currently
it's hard-coded in 'internal-complete-buffer':

#+begin_src c
  else if (EQ (flag, Qmetadata))
    return list3 (Qmetadata,
                  Fcons (Qcategory, Qbuffer),
                  Fcons (Qcycle_sort_function, Qidentity));
#+end_src

Re: Updating *Completions* as you type

[Spencer Baugh]

Juri Linkov <juri@linkov.net> writes:

>>> Actually there is already a nil value in completions-sort with the tag
>>> "No sorting".  This works nicely for 'C-x b'.  The remaining need is
>>> to be able to set it only for 'C-x b', not for other completion types.
>>
>> I feel quite strongly that we should not extend completions-sort to be
>> able to affect different completion types differently.  Instead I think
>> we should leave completions-sort as a blanket configuration for what to
>> do if the completion metadata does not specify display-sort-function,
>> and if we want to allow customizing an individual completion type, that
>> completion type should specify a display-sort-function which can be
>> customized.
>>
>> If we do extend completions-sort to affect different completion types
>> differently, I expect we'll see lots of silly things, like packages with
>> new completion types which automatically install changes to
>> completions-sort instead of just specifying their own
>> display-sort-function.
>
> Probably different completion types should provide separate options
> such as e.g. 'read-char-by-name-sort' defines 'display-sort-function'
> for 'read-char-by-name'.

Agreed.

> The same could be added for 'C-x b'.  But unfortunately currently
> it's hard-coded in 'internal-complete-buffer':
>
> #+begin_src c
>   else if (EQ (flag, Qmetadata))
>     return list3 (Qmetadata,
>                   Fcons (Qcategory, Qbuffer),
>                   Fcons (Qcycle_sort_function, Qidentity));
> #+end_src

Not hard to fix.  I will post a patch to make this customizable soon.

Re: Updating *Completions* as you type

[Spencer Baugh]

[-- Attachment #1: Type: text/plain, Size: 1697 bytes --]

Spencer Baugh <sbaugh@catern.com> writes:

> Juri Linkov <juri@linkov.net> writes:
>
>>>> Actually there is already a nil value in completions-sort with the tag
>>>> "No sorting".  This works nicely for 'C-x b'.  The remaining need is
>>>> to be able to set it only for 'C-x b', not for other completion types.
>>>
>>> I feel quite strongly that we should not extend completions-sort to be
>>> able to affect different completion types differently.  Instead I think
>>> we should leave completions-sort as a blanket configuration for what to
>>> do if the completion metadata does not specify display-sort-function,
>>> and if we want to allow customizing an individual completion type, that
>>> completion type should specify a display-sort-function which can be
>>> customized.
>>>
>>> If we do extend completions-sort to affect different completion types
>>> differently, I expect we'll see lots of silly things, like packages with
>>> new completion types which automatically install changes to
>>> completions-sort instead of just specifying their own
>>> display-sort-function.
>>
>> Probably different completion types should provide separate options
>> such as e.g. 'read-char-by-name-sort' defines 'display-sort-function'
>> for 'read-char-by-name'.
>
> Agreed.
>
>> The same could be added for 'C-x b'.  But unfortunately currently
>> it's hard-coded in 'internal-complete-buffer':
>>
>> #+begin_src c
>>   else if (EQ (flag, Qmetadata))
>>     return list3 (Qmetadata,
>>                   Fcons (Qcategory, Qbuffer),
>>                   Fcons (Qcycle_sort_function, Qidentity));
>> #+end_src
>
> Not hard to fix.  I will post a patch to make this customizable soon.

How about this?


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Add-read-buffer-sort.patch --]
[-- Type: text/x-patch, Size: 3293 bytes --]

From 2c5807193713893b8cecacf6ae9318e3e504fb93 Mon Sep 17 00:00:00 2001
From: Spencer Baugh <sbaugh@catern.com>
Date: Wed, 18 Oct 2023 22:28:59 -0400
Subject: [PATCH] Add read-buffer-sort

Add the ability to customize how read-buffer (actually
internal-complete-buffer) sorts the buffer completion candidates.

* lisp/cus-start.el (standard): Add customization information for
read-buffer-sort.
* src/minibuf.c (Finternal_complete_buffer): Use read-buffer-sort.
(syms_of_minibuf): Add read-buffer-sort.
---
 lisp/cus-start.el |  4 ++++
 src/minibuf.c     | 26 +++++++++++++++++++++++---
 2 files changed, 27 insertions(+), 3 deletions(-)

diff --git a/lisp/cus-start.el b/lisp/cus-start.el
index 054683d7cf6..569a0cf54b3 100644
--- a/lisp/cus-start.el
+++ b/lisp/cus-start.el
@@ -439,6 +439,10 @@ minibuffer-prompt-properties--setter
 	     (read-buffer-function minibuffer
 				   (choice (const nil)
 					   function))
+             (read-buffer-sort minibuffer
+                               (choice (const :tag "completions-sort controls sorting" nil)
+                                       (const :tag "sort matching buffer-list" buffer-list))
+                               "30.1")
 	     ;; msdos.c
 	     (dos-unsupported-char-glyph display integer)
 	     ;; nsterm.m
diff --git a/src/minibuf.c b/src/minibuf.c
index 58adde1bf66..715b4c08886 100644
--- a/src/minibuf.c
+++ b/src/minibuf.c
@@ -2186,9 +2186,13 @@ DEFUN ("internal-complete-buffer", Finternal_complete_buffer, Sinternal_complete
   else if (EQ (flag, Qlambda))
     return Ftest_completion (string, Vbuffer_alist, predicate);
   else if (EQ (flag, Qmetadata))
-    return list3 (Qmetadata,
-                  Fcons (Qcategory, Qbuffer),
-                  Fcons (Qcycle_sort_function, Qidentity));
+    {
+      Lisp_Object res = list2 (Fcons (Qcategory, Qbuffer),
+			       Fcons (Qcycle_sort_function, Qidentity));
+      if (EQ (Vread_buffer_sort, Qbuffer_list))
+	res = Fcons (Fcons (Qdisplay_sort_function, Qidentity), res);
+      return Fcons (Qmetadata, res);
+    }
   else
     return Qnil;
 }
@@ -2323,6 +2327,7 @@ syms_of_minibuf (void)
   DEFSYM (Qcase_fold_search, "case-fold-search");
   DEFSYM (Qmetadata, "metadata");
   DEFSYM (Qcycle_sort_function, "cycle-sort-function");
+  DEFSYM (Qdisplay_sort_function, "display-sort-function");
 
   /* A frame parameter.  */
   DEFSYM (Qminibuffer_exit, "minibuffer-exit");
@@ -2522,6 +2527,21 @@ syms_of_minibuf (void)
 the minibuffer.  However, if `minibuffer-restore-windows' is present
 in `minibuffer-exit-hook', exiting the minibuffer will remove the window
 showing the *Completions* buffer, if any.  */);
+
+  DEFVAR_LISP ("read-buffer-sort", Vread_buffer_sort,
+	       doc: /* Non-nil means sort completions in `read-buffer'.
+
+If this is nil (the default), completions in `read-buffer' are sorted
+according to `completions-sort'.
+
+If this is `buffer-list', completions are sorted to match the order of
+`buffer-list'.
+
+This variable only affects the default `read-buffer', so if
+`read-buffer-function' is set to a function which does not use
+`internal-complete-buffer', this variable will have no effect.*/);
+  Vread_buffer_sort = Qnil;
+
   read_minibuffer_restore_windows = true;
 
   defsubr (&Sactive_minibuffer_window);
-- 
2.41.0

Re: Updating *Completions* as you type

[Juri Linkov]

> How about this?

Looks good.

> +             (read-buffer-sort minibuffer
> +                               (choice (const :tag "completions-sort controls sorting" nil)
> +                                       (const :tag "sort matching buffer-list" buffer-list))
> +                               "30.1")

Shouldn't 'read-buffer-sort' support other choices from 'completions-sort'?
Your new choice 'historical' would sort buffers by the order of 'M-p'
the same way as 'buffer-list' will sort buffers by the order of 'M-n'.
Also 'alphabetical' would be useful for 'C-x b' when 'completions-sort'
was customized to something else.

Re: Updating *Completions* as you type

[sbaugh]

[-- Attachment #1: Type: text/plain, Size: 1322 bytes --]

Juri Linkov <juri@linkov.net> writes:
>> How about this?
>
> Looks good.
>
>> +             (read-buffer-sort minibuffer
>> +                               (choice (const :tag "completions-sort controls sorting" nil)
>> +                                       (const :tag "sort matching buffer-list" buffer-list))
>> +                               "30.1")
>
> Shouldn't 'read-buffer-sort' support other choices from 'completions-sort'?
> Your new choice 'historical' would sort buffers by the order of 'M-p'
> the same way as 'buffer-list' will sort buffers by the order of 'M-n'.
> Also 'alphabetical' would be useful for 'C-x b' when 'completions-sort'
> was customized to something else.

Finally got to it, I've added that in this patch (which combines both
the read-buffer-sort and completions-sort changes).

The nice reusable part is that the new minibuffer-sort-by-history and
minibuffer-sort-alphabetically should work for any completion table.  So
it will be easy to just add options for different completion tables,
e.g. read-file-name-sort, and they can just use those pre-existing
functions.

This does require adding the new dynamic variable
minibuffer-completion-base which maybe should go in some other place, or
could be derived from some other variable - I'm not sure.  Possibly it's
fine as I did it.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Add-read-buffer-sort-to-customize-how-read-buffer-so.patch --]
[-- Type: text/x-patch, Size: 8826 bytes --]

From 5e898c3ef323971af89b80fe37d94be0bf29443f Mon Sep 17 00:00:00 2001
From: Spencer Baugh <sbaugh@catern.com>
Date: Tue, 17 Oct 2023 09:09:55 -0400
Subject: [PATCH] Add read-buffer-sort to customize how read-buffer sorts
 completions

Add the ability to customize how read-buffer (actually
internal-complete-buffer) sorts the buffer completion candidates.

Also add minibuffer-sort-alphabetically and minibuffer-sort-by-history
to provide some reasonable options for read-buffer-sort.

And add support for them to completions-sort, too, as a simple change.

* lisp/minibuffer.el (completions-sort): Document 'historical option.
(minibuffer-completion-help): Support 'historical option.
(minibuffer-sort-alphabetically)
(minibuffer-completion-base, minibuffer-sort-by-history): Add.
* lisp/cus-start.el (standard): Add customization information for
read-buffer-sort.
* src/minibuf.c (Finternal_complete_buffer): Use read-buffer-sort.
(syms_of_minibuf): Add read-buffer-sort.
---
 lisp/cus-start.el  |  7 ++++++
 lisp/minibuffer.el | 63 +++++++++++++++++++++++++++++++++++++++++-----
 src/minibuf.c      | 32 ++++++++++++++++++++---
 3 files changed, 93 insertions(+), 9 deletions(-)

diff --git a/lisp/cus-start.el b/lisp/cus-start.el
index 6d83aaf4d14..40bc4078d77 100644
--- a/lisp/cus-start.el
+++ b/lisp/cus-start.el
@@ -441,6 +441,13 @@ minibuffer-prompt-properties--setter
 	     (read-buffer-function minibuffer
 				   (choice (const nil)
 					   function))
+             (read-buffer-sort minibuffer
+                               (choice (const :tag "completions-sort controls sorting" nil)
+                                       (const :tag "sort matching buffer-list" buffer-list)
+                                       (const :tag "sort alphabetically" minibuffer-sort-alphabetically)
+                                       (const :tag "sort historically" minibuffer-sort-by-history)
+                                       (function :tag "User-defined function"))
+                               "30.1")
 	     ;; msdos.c
 	     (dos-unsupported-char-glyph display integer)
 	     ;; nsterm.m
diff --git a/lisp/minibuffer.el b/lisp/minibuffer.el
index 026613c9eb2..49086ff6eff 100644
--- a/lisp/minibuffer.el
+++ b/lisp/minibuffer.el
@@ -1307,14 +1307,26 @@ completion-cycle-threshold
 (defcustom completions-sort 'alphabetical
   "Sort candidates in the *Completions* buffer.
 
-The value can be nil to disable sorting, `alphabetical' for
-alphabetical sorting or a custom sorting function.  The sorting
-function takes and returns a list of completion candidate
-strings."
+Candidate completions in the *Completions* are sorted depending
+on the value.
+
+If nil, sorting is disabled.
+If `alphabetical', candidates are sorted by
+`minibuffer-sort-alphabetically'.
+If `historical', candidates are sorted by
+`minibuffer-sort-by-history'.
+If a function, the function is called to sort the candidates.
+The sorting function takes and returns a list of completion
+candidate strings.
+
+If the completion-specific metadata provides a
+`display-sort-function', that is used instead and this value is
+ignored."
   :type '(choice (const :tag "No sorting" nil)
                  (const :tag "Alphabetical sorting" alphabetical)
+                 (const :tag "Historical sorting" historical)
                  (function :tag "Custom function"))
-  :version "29.1")
+  :version "30.1")
 
 (defcustom completions-group nil
   "Enable grouping of completion candidates in the *Completions* buffer.
@@ -1640,6 +1652,43 @@ minibuffer--sort-preprocess-history
                      (substring c base-size)))
                  hist)))))
 
+(defun minibuffer-sort-alphabetically (completions)
+  "Sort COMPLETIONS alphabetically.
+
+COMPLETIONS are sorted alphabetically by `string-lessp'.
+
+This is a suitable function to use for `completions-sort' or to
+include as `display-sort-function' in completion metadata."
+  (sort completions #'string-lessp))
+
+(defvar minibuffer-completion-base nil
+  "The base for the current completion.
+
+This is the part of the current minibuffer input which is not
+being completed on.  This is primarily relevant for file names,
+where this is the directory component of the file name.")
+
+(defun minibuffer-sort-by-history (completions)
+  "Sort COMPLETIONS by their position in `minibuffer-history-variable'.
+
+COMPLETIONS are sorted first by `minibuffer-sort-alphbetically',
+then any elements occuring in the minibuffer history list are
+moved to the front based on the order they occur in the history.
+If a history variable hasn't been specified for this call of
+`completing-read', COMPLETIONS are sorted only by
+`minibuffer-sort-alphbetically'.
+
+This is a suitable function to use for `completions-sort' or to
+include as `display-sort-function' in completion metadata."
+  (let ((alphabetized (sort completions #'string-lessp)))
+    ;; Only use history when it's specific to these completions.
+    (if (eq minibuffer-history-variable
+            (default-value minibuffer-history-variable))
+        alphabetized
+      (minibuffer--sort-by-position
+       (minibuffer--sort-preprocess-history minibuffer-completion-base)
+       alphabetized))))
+
 (defun minibuffer--group-by (group-fun sort-fun elems)
   "Group ELEMS by GROUP-FUN and sort groups by SORT-FUN."
   (let ((groups))
@@ -2470,6 +2519,7 @@ minibuffer-completion-help
       (let* ((last (last completions))
              (base-size (or (cdr last) 0))
              (prefix (unless (zerop base-size) (substring string 0 base-size)))
+             (minibuffer-completion-base (substring string 0 base-size))
              (base-prefix (buffer-substring (minibuffer--completion-prompt-end)
                                             (+ start base-size)))
              (base-suffix
@@ -2540,7 +2590,8 @@ minibuffer-completion-help
                                             (funcall sort-fun completions)
                                           (pcase completions-sort
                                             ('nil completions)
-                                            ('alphabetical (sort completions #'string-lessp))
+                                            ('alphabetical (minibuffer-sort-alphabetically completions))
+                                            ('historical (minibuffer-sort-by-history completions))
                                             (_ (funcall completions-sort completions)))))
 
                       ;; After sorting, group the candidates using the
diff --git a/src/minibuf.c b/src/minibuf.c
index 58adde1bf66..04551afeb79 100644
--- a/src/minibuf.c
+++ b/src/minibuf.c
@@ -2186,9 +2186,15 @@ DEFUN ("internal-complete-buffer", Finternal_complete_buffer, Sinternal_complete
   else if (EQ (flag, Qlambda))
     return Ftest_completion (string, Vbuffer_alist, predicate);
   else if (EQ (flag, Qmetadata))
-    return list3 (Qmetadata,
-                  Fcons (Qcategory, Qbuffer),
-                  Fcons (Qcycle_sort_function, Qidentity));
+    {
+      Lisp_Object res = list2 (Fcons (Qcategory, Qbuffer),
+			       Fcons (Qcycle_sort_function, Qidentity));
+      if (EQ (Vread_buffer_sort, Qbuffer_list))
+	res = Fcons (Fcons (Qdisplay_sort_function, Qidentity), res);
+      else if (FUNCTIONP (Vread_buffer_sort))
+	res = Fcons (Fcons (Qdisplay_sort_function, Vread_buffer_sort), res);
+      return Fcons (Qmetadata, res);
+    }
   else
     return Qnil;
 }
@@ -2323,6 +2329,7 @@ syms_of_minibuf (void)
   DEFSYM (Qcase_fold_search, "case-fold-search");
   DEFSYM (Qmetadata, "metadata");
   DEFSYM (Qcycle_sort_function, "cycle-sort-function");
+  DEFSYM (Qdisplay_sort_function, "display-sort-function");
 
   /* A frame parameter.  */
   DEFSYM (Qminibuffer_exit, "minibuffer-exit");
@@ -2522,6 +2529,25 @@ syms_of_minibuf (void)
 the minibuffer.  However, if `minibuffer-restore-windows' is present
 in `minibuffer-exit-hook', exiting the minibuffer will remove the window
 showing the *Completions* buffer, if any.  */);
+
+  DEFVAR_LISP ("read-buffer-sort", Vread_buffer_sort,
+	       doc: /* Non-nil means sort completions in `read-buffer'.
+
+If this is nil (the default), completions in `read-buffer' are sorted
+according to `completions-sort'.
+
+If this is `buffer-list', completions are sorted to match the order of
+`buffer-list'.
+
+If a function, the function is called to sort the candidates.  Some
+possible values are `minibuffer-sort-alphabetically' and
+`minibuffer-sort-by-history'.
+
+This variable only affects the default `read-buffer', so if
+`read-buffer-function' is set to a function which does not use
+`internal-complete-buffer', this variable will have no effect.*/);
+  Vread_buffer_sort = Qnil;
+
   read_minibuffer_restore_windows = true;
 
   defsubr (&Sactive_minibuffer_window);
-- 
2.42.1

Re: Updating *Completions* as you type

[Juri Linkov]

> Finally got to it, I've added that in this patch (which combines both
> the read-buffer-sort and completions-sort changes).

Thanks, a quick test shows that everything works nicely.

> +      Lisp_Object res = list2 (Fcons (Qcategory, Qbuffer),
> +			       Fcons (Qcycle_sort_function, Qidentity));
> +      if (EQ (Vread_buffer_sort, Qbuffer_list))
> +	res = Fcons (Fcons (Qdisplay_sort_function, Qidentity), res);
> +      else if (FUNCTIONP (Vread_buffer_sort))
> +	res = Fcons (Fcons (Qdisplay_sort_function, Vread_buffer_sort), res);
> +      return Fcons (Qmetadata, res);

I still have doubts about adding separate sorting options for
every completion category.

What do you think about adding customization of display-sort-function
to the existing option completion-category-overrides?

This would allow users such customization for 'C-x b' and 'C-x p p'
to sort buffers and projects by history:

  (setopt completion-category-overrides
    '((buffer (display-sort-function . minibuffer-sort-by-history))
      (project-file (display-sort-function . minibuffer-sort-by-history))

And default values could be specified in completion-category-defaults.

Re: Updating *Completions* as you type

[Spencer Baugh]

Juri Linkov <juri@linkov.net> writes:
>> Finally got to it, I've added that in this patch (which combines both
>> the read-buffer-sort and completions-sort changes).
>
> Thanks, a quick test shows that everything works nicely.
>
>> +      Lisp_Object res = list2 (Fcons (Qcategory, Qbuffer),
>> +			       Fcons (Qcycle_sort_function, Qidentity));
>> +      if (EQ (Vread_buffer_sort, Qbuffer_list))
>> +	res = Fcons (Fcons (Qdisplay_sort_function, Qidentity), res);
>> +      else if (FUNCTIONP (Vread_buffer_sort))
>> +	res = Fcons (Fcons (Qdisplay_sort_function, Vread_buffer_sort), res);
>> +      return Fcons (Qmetadata, res);
>
> I still have doubts about adding separate sorting options for
> every completion category.

Not just every completion category, every completion table.

What are your doubts?  I don't think it ends up being too many options
in practice.  The discoverability is worse than a centralized solution,
but it also keeps configuration localized to the individual component,
which is nice and makes it easier to group buffer/file/etc settings
together in configuration and the Customize interface.

One important feature of separate sorting options is that they support
sorting details which are specific to the individual completion table.
For buffers, for example, sorting by the buffer-list can be done by just
setting display-sort-function to identity, but we probably shouldn't
expose that fact to the user.  (We could expose a generic "sort by
defaults" but that would be less efficient.)

The sorting style might even require the sorting option to actually
change the completion table's behavior.  File name completion could be
sorted by mtime, for example, by having the completion table include
mtime as a text property when read-file-name-sort=mtime, and then
setting display-sort-function to something which reads that property.  I
think that can only work if we have separate sorting options.

> What do you think about adding customization of display-sort-function
> to the existing option completion-category-overrides?
>
> This would allow users such customization for 'C-x b' and 'C-x p p'
> to sort buffers and projects by history:
>
>   (setopt completion-category-overrides
>     '((buffer (display-sort-function . minibuffer-sort-by-history))
>       (project-file (display-sort-function . minibuffer-sort-by-history))
>
> And default values could be specified in completion-category-defaults.

Hm, this could be okay.  It does sacrifice the extra level of
customization possible with separate sorting options that I just
described.

I think the default values would be part of the completion table itself;
a completion table can just set display-sort-function itself, it doesn't
need completion-category-defaults to do that.

Indeed, currently completion-category-overrides and defaults are used
only to configure things which can't be directly set by the completion
table.  That's my understanding of the purpose of completion categories:
the completion table returns a category symbol as a somewhat roundabout
way to configure completion-styles and completion-cycle-threshold to
values which are specific to the table.

But the completion table is already able to set display-sort-function on
its own.  So it's novel to also have completion-category-overrides able
to do that.

Maybe we should decide what things are supposed to be configured by
category and what things are supposed to be configured by the table
itself.  Because in theory we could allow setting completion-styles and
completion-cycle-threshold with completion table metadata, and I'm not
sure why we didn't do it that way - it could even be nice to have a
completion style which is specific to an individual completion table.

Re: Updating *Completions* as you type

[Juri Linkov]

> For buffers, for example, sorting by the buffer-list can be done by just
> setting display-sort-function to identity, but we probably shouldn't
> expose that fact to the user.  (We could expose a generic "sort by
> defaults" but that would be less efficient.)

It should be fine to use 'identity' as a value when the Customization UI
will show a more meaningful tag like "Unsorted".

> The sorting style might even require the sorting option to actually
> change the completion table's behavior.  File name completion could be
> sorted by mtime, for example, by having the completion table include
> mtime as a text property when read-file-name-sort=mtime, and then
> setting display-sort-function to something which reads that property.  I
> think that can only work if we have separate sorting options.

Also a separate option might be needed when a completion table
supports a non-standard sorting order such as
'mule--ucs-names-sort-by-code' when 'read-char-by-name-sort'
is customized to 'code'.  But the users still should have freedom
to override all metadata: display-sort-function, affixation-function,
group-function, etc.

>>   (setopt completion-category-overrides
>>     '((buffer (display-sort-function . minibuffer-sort-by-history))
>>       (project-file (display-sort-function . minibuffer-sort-by-history))
>>
>> And default values could be specified in completion-category-defaults.
>
> I think the default values would be part of the completion table itself;
> a completion table can just set display-sort-function itself, it doesn't
> need completion-category-defaults to do that.

OTOH, the completion table needs only to return its category in metadata.
Otherwise every completion table of the same category would need to
duplicate display-sort-function in its completing-read.

> Maybe we should decide what things are supposed to be configured by
> category and what things are supposed to be configured by the table
> itself.  Because in theory we could allow setting completion-styles and
> completion-cycle-threshold with completion table metadata, and I'm not
> sure why we didn't do it that way - it could even be nice to have a
> completion style which is specific to an individual completion table.

I guess when an individual completion table needs to use metadata
slightly different from other similar cases then it should provide
a different category/subcategory.

Re: Updating *Completions* as you type

[Spencer Baugh]

Juri Linkov <juri@linkov.net> writes:
>> For buffers, for example, sorting by the buffer-list can be done by just
>> setting display-sort-function to identity, but we probably shouldn't
>> expose that fact to the user.  (We could expose a generic "sort by
>> defaults" but that would be less efficient.)
>
> It should be fine to use 'identity' as a value when the Customization UI
> will show a more meaningful tag like "Unsorted".

Yes, that's fine.  But "Unsorted" is different from "Sorted by
buffer-list", which is the behavior one gets with read-buffer.  In other
words, the documentation can be more specific for read-buffer-sort.

>> The sorting style might even require the sorting option to actually
>> change the completion table's behavior.  File name completion could be
>> sorted by mtime, for example, by having the completion table include
>> mtime as a text property when read-file-name-sort=mtime, and then
>> setting display-sort-function to something which reads that property.  I
>> think that can only work if we have separate sorting options.
>
> Also a separate option might be needed when a completion table
> supports a non-standard sorting order such as
> 'mule--ucs-names-sort-by-code' when 'read-char-by-name-sort'
> is customized to 'code'.  But the users still should have freedom
> to override all metadata: display-sort-function, affixation-function,
> group-function, etc.
>
>>>   (setopt completion-category-overrides
>>>     '((buffer (display-sort-function . minibuffer-sort-by-history))
>>>       (project-file (display-sort-function . minibuffer-sort-by-history))
>>>
>>> And default values could be specified in completion-category-defaults.
>>
>> I think the default values would be part of the completion table itself;
>> a completion table can just set display-sort-function itself, it doesn't
>> need completion-category-defaults to do that.
>
> OTOH, the completion table needs only to return its category in metadata.
> Otherwise every completion table of the same category would need to
> duplicate display-sort-function in its completing-read.

Interesting point.  In some sense, the category gives us an inheritance
mechanism for completion tables, where a completion table can just use
the defaults for its category rather than specifying all the details.
But if the completion table has some specific sorting function it wants,
it can provide that by explicitly returning it in the metadata.

And completion-category-overrides gives the user a way to do
customization of the category defaults.

So perhaps, for a given metadata, display-sort-function would be:

(or
 (alist-get 'display-sort-function metadata)
 (alist-get 'display-sort-function (alist-get category completion-category-overrides))
 (alist-get 'display-sort-function (alist-get category completion-category-defaults))
 completions-sort)

That makes sense to me.

And (alist-get 'display-sort-function metadata) is configurable with
separate completion-table-specific options, like read-buffer-sort.

If we do this, completions-sort acts like a "default" entry in
completion-category-defaults, which acts for categories which aren't
otherwise matched.  Which makes a lot of sense, and makes it behave like
completion-styles and completion-cycle-threshold.

I expect some disagreement about the following question: should
completion-category-overrides override the display-sort-function
returned by the completion table?  That is, should it instead be:

(or
 (alist-get 'display-sort-function (alist-get category completion-category-overrides))
 (alist-get 'display-sort-function metadata)
 (alist-get 'display-sort-function (alist-get category completion-category-defaults))
 completions-sort)

I think no.  The most specific customization (customizing an individual
completion table) should override more broad customizations (customizing
an entire category).  I don't think there's a reasonable use-case for
overriding a display-sort-function explicitly returned by a table - we
can just make that display-sort-function configurable directly.

We probably will want to make this clear in the documentation, of
course.

>> Maybe we should decide what things are supposed to be configured by
>> category and what things are supposed to be configured by the table
>> itself.  Because in theory we could allow setting completion-styles and
>> completion-cycle-threshold with completion table metadata, and I'm not
>> sure why we didn't do it that way - it could even be nice to have a
>> completion style which is specific to an individual completion table.
>
> I guess when an individual completion table needs to use metadata
> slightly different from other similar cases then it should provide
> a different category/subcategory.

Or the individual completion table just returns that slightly different
metadata itself, and it overrides the metadata pulled from its category.

Re: Updating *Completions* as you type

[Juri Linkov]

> I expect some disagreement about the following question: should
> completion-category-overrides override the display-sort-function
> returned by the completion table?  That is, should it instead be:
>
> (or
>  (alist-get 'display-sort-function (alist-get category completion-category-overrides))
>  (alist-get 'display-sort-function metadata)
>  (alist-get 'display-sort-function (alist-get category completion-category-defaults))
>  completions-sort)

I think this is the most correct precedence since the users should be
able to override the function call metadata.  We have an analogous
priority levels for 'display-buffer':

1. display-buffer-overriding-action
2. display-buffer-alist
3. function call arguments that correspond to completion metadata
4. display-buffer-base-action
5. display-buffer-fallback-action

Since completion-category-overrides is a user option
it corresponds to display-buffer-alist.
And completion-category-defaults looks like
display-buffer-base-action.

Re: Updating *Completions* as you type

[sbaugh]

Juri Linkov <juri@linkov.net> writes:
>> I expect some disagreement about the following question: should
>> completion-category-overrides override the display-sort-function
>> returned by the completion table?  That is, should it instead be:
>>
>> (or
>>  (alist-get 'display-sort-function (alist-get category completion-category-overrides))
>>  (alist-get 'display-sort-function metadata)
>>  (alist-get 'display-sort-function (alist-get category completion-category-defaults))
>>  completions-sort)
>
> I think this is the most correct precedence since the users should be
> able to override the function call metadata.  We have an analogous
> priority levels for 'display-buffer':
>
> 1. display-buffer-overriding-action
> 2. display-buffer-alist
> 3. function call arguments that correspond to completion metadata
> 4. display-buffer-base-action
> 5. display-buffer-fallback-action
>
> Since completion-category-overrides is a user option
> it corresponds to display-buffer-alist.
> And completion-category-defaults looks like
> display-buffer-base-action.

A few points in favor of

  (alist-get 'display-sort-function metadata)
  (alist-get 'display-sort-function (alist-get category completion-category-overrides))

instead:

- Again, the user is still able to configure the display-sort-function
  by configuring the individual completion table.  That's not true with
  display-buffer.  The only configuration mechanism is
  display-buffer-alist.

- Given that the user can still configure the display-sort-function, I
  don't see any use case where the user should override it.  (A buggy
  completion table that returns the wrong display-sort-function?  But
  that should just be fixed.)

- display-buffer-alist is driven purely by buffer-match-p conditions, so
  there's a linear sequence of overriding.  The display-sort-function
  has two levels: the completion table and the completion category.
  Since the table is more specific than the category, it should override
  the category.

- As a minor point, I, and many other Emacs users IME, find the
  display-buffer configuration to be complex and hard to use, so I don't
  think we should try to emulate it too much.

- The "overrides" in "completion-category-overrides" is just a name.  It
  doesn't mean that we have to make it override everything else.

I suppose we could add a new completion-category-table-overrides, if we
find a use case for making the category override the table but it's
rare.  That's starting to reach the complexity of display-buffer-alist
though...

Alternatively, could we just add support for configuring the individual
table now, and add category-based configuration later?  We don't need to
add everything all at once, and that will give us valuable user
feedback.

Re: Updating *Completions* as you type

[Juri Linkov]

>>> (or
>>>  (alist-get 'display-sort-function (alist-get category completion-category-overrides))
>>>  (alist-get 'display-sort-function metadata)
>>>  (alist-get 'display-sort-function (alist-get category completion-category-defaults))
>>>  completions-sort)
>>
>> 1. display-buffer-overriding-action
>> 2. display-buffer-alist
>> 3. function call arguments that correspond to completion metadata
>> 4. display-buffer-base-action
>> 5. display-buffer-fallback-action
>
> A few points in favor of
>
>   (alist-get 'display-sort-function metadata)
>   (alist-get 'display-sort-function (alist-get category completion-category-overrides))
>
> instead:
>
> - Again, the user is still able to configure the display-sort-function
>   by configuring the individual completion table.

Does this mean that every individual completion table should have
a separate user option?

>   That's not true with display-buffer.  The only configuration
>   mechanism is display-buffer-alist.

Actually several user options already exist for display-buffer such as
display-comint-buffer-action and display-tex-shell-buffer-action.
They cover a whole category of display-buffer calls that display
all types of comint buffers, etc.  So these options correspond
to completion categories, not to individual completion tables.

> - Given that the user can still configure the display-sort-function, I
>   don't see any use case where the user should override it.  (A buggy
>   completion table that returns the wrong display-sort-function?  But
>   that should just be fixed.)

I don't understand how the user can configure the display-sort-function,
please elaborate.

> - display-buffer-alist is driven purely by buffer-match-p conditions, so
>   there's a linear sequence of overriding.  The display-sort-function
>   has two levels: the completion table and the completion category.
>   Since the table is more specific than the category, it should override
>   the category.

Agreed.  But this means only one thing that a user option
for an individual completion table should take precedence
over completion-category-overrides.

> - As a minor point, I, and many other Emacs users IME, find the
>   display-buffer configuration to be complex and hard to use, so I don't
>   think we should try to emulate it too much.

What is an alternative?  A myriad of individual options?

> - The "overrides" in "completion-category-overrides" is just a name.  It
>   doesn't mean that we have to make it override everything else.

Why not to do what the name suggests?

> Alternatively, could we just add support for configuring the individual
> table now, and add category-based configuration later?  We don't need to
> add everything all at once, and that will give us valuable user
> feedback.

I see no need to add individual options as all.  Every completion table
that significantly differs from other tables so that it needs a separate
display-sort-function, could provide a separate category.  For example,
there is a category 'buffer'.  If 'switch-to-buffer' needs another
display-sort-function it could provide a category 'buffer-for-switching'.

Re: Updating *Completions* as you type

[Spencer Baugh]

Juri Linkov <juri@linkov.net> writes:
>>>> (or
>>>>  (alist-get 'display-sort-function (alist-get category completion-category-overrides))
>>>>  (alist-get 'display-sort-function metadata)
>>>>  (alist-get 'display-sort-function (alist-get category completion-category-defaults))
>>>>  completions-sort)
>>>
>>> 1. display-buffer-overriding-action
>>> 2. display-buffer-alist
>>> 3. function call arguments that correspond to completion metadata
>>> 4. display-buffer-base-action
>>> 5. display-buffer-fallback-action
>>
>> A few points in favor of
>>
>>   (alist-get 'display-sort-function metadata)
>>   (alist-get 'display-sort-function (alist-get category completion-category-overrides))
>>
>> instead:
>>
>> - Again, the user is still able to configure the display-sort-function
>>   by configuring the individual completion table.
>
> Does this mean that every individual completion table should have
> a separate user option?

No: only the completion tables which specify a display-sort-function in
their metadata.  All such completion tables should have a user option to
configure that display-sort-function.

>>   That's not true with display-buffer.  The only configuration
>>   mechanism is display-buffer-alist.
>
> Actually several user options already exist for display-buffer such as
> display-comint-buffer-action and display-tex-shell-buffer-action.
> They cover a whole category of display-buffer calls that display
> all types of comint buffers, etc.  So these options correspond
> to completion categories, not to individual completion tables.

Interesting.

It looks like those user options configure a broad category, as you say,
and they do this by changing the ACTION function call argument in
various places.

So, those user options configure a broad category, and then more
specific configuration overrides that broad category.  I suppose that's
what I also suggest for completion sorting.

>> - Given that the user can still configure the display-sort-function, I
>>   don't see any use case where the user should override it.  (A buggy
>>   completion table that returns the wrong display-sort-function?  But
>>   that should just be fixed.)
>
> I don't understand how the user can configure the display-sort-function,
> please elaborate.

If a completion table specifies a display-sort-function in its metadata,
it should have a user option to configure that display-sort-function.

>> - display-buffer-alist is driven purely by buffer-match-p conditions, so
>>   there's a linear sequence of overriding.  The display-sort-function
>>   has two levels: the completion table and the completion category.
>>   Since the table is more specific than the category, it should override
>>   the category.
>
> Agreed.  But this means only one thing that a user option
> for an individual completion table should take precedence
> over completion-category-overrides.

Well, yes.  So then we agree that a user option for an individual
completion table, if it exists, should take precedence over
completion-category-overrides?  So then we're only disagreeing over
whether such options should exist?

>> - As a minor point, I, and many other Emacs users IME, find the
>>   display-buffer configuration to be complex and hard to use, so I don't
>>   think we should try to emulate it too much.
>
> What is an alternative?  A myriad of individual options?

For display-buffer configuration?  I'm not sure, I don't want to get
into it too much.

For completion sorting configuration, though, I'm not suggesting adding
many individual options.  I'm quite happy with the proposal of,
"categories can determine the display-sort-function, and then options
for individual completion tables will override that".

In that world, there would be a read-buffer-sort option - but it would
be nil by default, so the read-buffer completion table by default
wouldn't specify a display-sort-function.  And the user could configure
read-buffer completion sorting using completion-category-overrides
however they like; and if they set read-buffer-sort to something else,
then it will in turn override the category-based configuration.

These individual options would also provide a natural place to document
behavior like "if you configure the display-sort-function for buffer
completion to 'identity, then the buffer sort order will match
(buffer-list)".  But the user could still make use of that information
by configuring the category.

>> Alternatively, could we just add support for configuring the individual
>> table now, and add category-based configuration later?  We don't need to
>> add everything all at once, and that will give us valuable user
>> feedback.
>
> I see no need to add individual options as all.  Every completion table
> that significantly differs from other tables so that it needs a separate
> display-sort-function, could provide a separate category.  For example,
> there is a category 'buffer'.  If 'switch-to-buffer' needs another
> display-sort-function it could provide a category 'buffer-for-switching'.

That won't work with the scenario I described before with sorting
file-name completion by mtime, where changing the sorting requires also
changing the completion table.

Also, this would require adding a category for essentially every
completion table.  For example, I see that read-from-kill-ring specifies
a display-sort-function, currently set to 'identity.  If we wanted to
make that configurable, it seems much easier to just do

          (if (eq action 'metadata)
              ;; Keep sorted by recency
-             '(metadata (display-sort-function . identity))
+             `(metadata (display-sort-function . ,read-from-kill-ring-sort))
            (complete-with-action action completions string pred)))

instead of adding a category 'kill-ring and sticking that into
completion-category-defaults.

Re: Updating *Completions* as you type

[Juri Linkov]

>>> - Again, the user is still able to configure the display-sort-function
>>>   by configuring the individual completion table.
>>
>> Does this mean that every individual completion table should have
>> a separate user option?
>
> No: only the completion tables which specify a display-sort-function in
> their metadata.  All such completion tables should have a user option to
> configure that display-sort-function.

How then users could change the sorting order for individual tables
that don't specify a display-sort-function to use an order different
from completions-sort?

> Well, yes.  So then we agree that a user option for an individual
> completion table, if it exists, should take precedence over
> completion-category-overrides?

The problem is that we can't distinguish two cases:

1. when display-sort-function is hard-coded in metadata
   by the author of the completion table;
2. when display-sort-function in metadata
   gets the value from the user option.

Since we can't distinguish these cases, then it makes more sense
when completion-category-overrides overrides everything:

  (alist-get 'display-sort-function (alist-get category completion-category-overrides))
  (alist-get 'display-sort-function metadata) ;; metadata with/out individual options
  (alist-get 'display-sort-function (alist-get category completion-category-defaults))

There is no problem with this because completion-category-overrides
is a user option as well, so everything still is under user control.

> So then we're only disagreeing over whether such options should exist?

Yes, I think we should add individual options only in exceptional cases.

> These individual options would also provide a natural place to document
> behavior like "if you configure the display-sort-function for buffer
> completion to 'identity, then the buffer sort order will match
> (buffer-list)".  But the user could still make use of that information
> by configuring the category.

I agree that an option with documentation could help in such cases
when a non-trivial sorting function is provided for a completion table.

>> I see no need to add individual options as all.  Every completion table
>> that significantly differs from other tables so that it needs a separate
>> display-sort-function, could provide a separate category.  For example,
>> there is a category 'buffer'.  If 'switch-to-buffer' needs another
>> display-sort-function it could provide a category 'buffer-for-switching'.
>
> That won't work with the scenario I described before with sorting
> file-name completion by mtime, where changing the sorting requires also
> changing the completion table.

I agree that individual options are required in such rare cases when
their values affect the completion table and its formatting.

> Also, this would require adding a category for essentially every
> completion table.  For example, I see that read-from-kill-ring specifies
> a display-sort-function, currently set to 'identity.

It's much simpler to add an extra line with a category.

> If we wanted to make that configurable, it seems much easier to just do
>
>           (if (eq action 'metadata)
>               ;; Keep sorted by recency
> -             '(metadata (display-sort-function . identity))
> +             `(metadata (display-sort-function . ,read-from-kill-ring-sort))
>             (complete-with-action action completions string pred)))

This is an incomplete patch, there should be also a dozen of lines
with defcustom, its docstring, the version number and a list
of choices, etc.  And all this for a very small percent of users
who would like to change this order.  This is too wasteful.
It would be much more efficient to allow doing the same
by customizing completion-category-overrides.

Re: Updating *Completions* as you type

[Spencer Baugh]

Juri Linkov <juri@linkov.net> writes:
>>>> - Again, the user is still able to configure the display-sort-function
>>>>   by configuring the individual completion table.
>>>
>>> Does this mean that every individual completion table should have
>>> a separate user option?
>>
>> No: only the completion tables which specify a display-sort-function in
>> their metadata.  All such completion tables should have a user option to
>> configure that display-sort-function.
>
> How then users could change the sorting order for individual tables
> that don't specify a display-sort-function to use an order different
> from completions-sort?

They can use the category if the table specifies one.

If the table neither specifies a category nor provides a table-specific
option, the display sort function for that table isn't currently
configurable.  Which I think we're both fine with?

So: we already accept that for some completion tables, it won't be
possible to customize their display sort function out of the box.

>> Well, yes.  So then we agree that a user option for an individual
>> completion table, if it exists, should take precedence over
>> completion-category-overrides?
>
> The problem is that we can't distinguish two cases:
>
> 1. when display-sort-function is hard-coded in metadata
>    by the author of the completion table;
> 2. when display-sort-function in metadata
>    gets the value from the user option.

I think we should just eliminate any instances of case 1.

Case 1 just means the completion table's display sort function isn't
currently configurable.  Which I think we've already accepted will be
the case for some tables, until we go and make them configurable either
by adding a table-specific option, adding a category, or both.

> Since we can't distinguish these cases, then it makes more sense
> when completion-category-overrides overrides everything:
>
>   (alist-get 'display-sort-function (alist-get category completion-category-overrides))
>   (alist-get 'display-sort-function metadata) ;; metadata with/out individual options
>   (alist-get 'display-sort-function (alist-get category completion-category-defaults))
>
> There is no problem with this because completion-category-overrides
> is a user option as well, so everything still is under user control.

Only if the completion table specifies a category.  Which most do not.

So we'd need to change it to specify a category.  And if we're doing
that, we could also change it to not hard-code a display-sort-function
at the same time.

That is, for any tables where the display-sort-function is currently
hardcoded, we can add a category, and remove the hardcoded
display-sort-function from the table metadata, and add the
display-sort-function to completion-category-defaults.

>> So then we're only disagreeing over whether such options should exist?
>
> Yes, I think we should add individual options only in exceptional cases.
>
>> These individual options would also provide a natural place to document
>> behavior like "if you configure the display-sort-function for buffer
>> completion to 'identity, then the buffer sort order will match
>> (buffer-list)".  But the user could still make use of that information
>> by configuring the category.
>
> I agree that an option with documentation could help in such cases
> when a non-trivial sorting function is provided for a completion table.
>
>>> I see no need to add individual options as all.  Every completion table
>>> that significantly differs from other tables so that it needs a separate
>>> display-sort-function, could provide a separate category.  For example,
>>> there is a category 'buffer'.  If 'switch-to-buffer' needs another
>>> display-sort-function it could provide a category 'buffer-for-switching'.
>>
>> That won't work with the scenario I described before with sorting
>> file-name completion by mtime, where changing the sorting requires also
>> changing the completion table.
>
> I agree that individual options are required in such rare cases when
> their values affect the completion table and its formatting.

OK, I think I can agree with that, if we agree that in those rare cases,
the individual options should take precedence over the category-based
configuration.

>> Also, this would require adding a category for essentially every
>> completion table.  For example, I see that read-from-kill-ring specifies
>> a display-sort-function, currently set to 'identity.
>
> It's much simpler to add an extra line with a category.
>
>> If we wanted to make that configurable, it seems much easier to just do
>>
>>           (if (eq action 'metadata)
>>               ;; Keep sorted by recency
>> -             '(metadata (display-sort-function . identity))
>> +             `(metadata (display-sort-function . ,read-from-kill-ring-sort))
>>             (complete-with-action action completions string pred)))
>
> This is an incomplete patch, there should be also a dozen of lines
> with defcustom, its docstring, the version number and a list
> of choices, etc.  And all this for a very small percent of users
> who would like to change this order.  This is too wasteful.
> It would be much more efficient to allow doing the same
> by customizing completion-category-overrides.

The docstring and list of choices for read-from-kill-ring-sort are
something we want anyway - we would like to document that 'identity for
read-for-kill-ring keeps the kill ring sorted by recency, for example.
I see no better place to document that.

The version number is also something we want anyway: if we just add a
category to read-from-kill-ring in Emacs 30, this will work only in
Emacs 30 and not in Emacs 29, and there's no way for a user to know that
other than by reading NEWS.

For such tables, I see three good possibilities (in order of my own
preference):

A.
- Add read-from-kill-ring-sort defaulting to identity (with docstring)
(diff is 1 line + defcustom)

B.
- Add read-from-kill-ring-sort defaulting to nil (with docstring)
- add the 'read-kill category to the metadata
- add 'read-kill to completion-category-defaults
(diff is 3 lines + defcustom)

C.
- Remove display-sort-function from the metadata
- add the 'read-kill category to the metadata
- add 'read-kill to completion-category-defaults
(diff is 3 lines)

If you really don't want the defcustom and associated documentation, I'm
OK with C.

The option which I think is not good is:

D.
- add the 'read-kill category to the metadata
- make completion-category-overrides take precedence over what is
  specified in the table metadata

(diff is 1 line)

This is a slightly smaller diff than option C, but I think it's a
fundamentally worse approach than C, because in the rare cases where we
do want an individual option for the table, we won't have a way for that
option to take precedence over the broader category-based configuration.

Re: Updating *Completions* as you type

[Juri Linkov]

>>>>> - Again, the user is still able to configure the display-sort-function
>>>>>   by configuring the individual completion table.
>>>>
>>>> Does this mean that every individual completion table should have
>>>> a separate user option?
>>>
>>> No: only the completion tables which specify a display-sort-function in
>>> their metadata.  All such completion tables should have a user option to
>>> configure that display-sort-function.
>>
>> How then users could change the sorting order for individual tables
>> that don't specify a display-sort-function to use an order different
>> from completions-sort?
>
> They can use the category if the table specifies one.
>
> If the table neither specifies a category nor provides a table-specific
> option, the display sort function for that table isn't currently
> configurable.  Which I think we're both fine with?

I think we should gradually add a category to most completion tables
to make them completely configurable, not just with display-sort-function,
but with all possible metadata.

Adding a category resembles a long-lasting and still ongoing process of
adding specific minibuffer history as a symbol to the HIST argument
of different calls of read-from-minibuffer.

> So: we already accept that for some completion tables, it won't be
> possible to customize their display sort function out of the box.
>
>>> Well, yes.  So then we agree that a user option for an individual
>>> completion table, if it exists, should take precedence over
>>> completion-category-overrides?
>>
>> The problem is that we can't distinguish two cases:
>>
>> 1. when display-sort-function is hard-coded in metadata
>>    by the author of the completion table;
>> 2. when display-sort-function in metadata
>>    gets the value from the user option.
>
> I think we should just eliminate any instances of case 1.

I don't think this is realistic to add an individual option in all cases.

> Case 1 just means the completion table's display sort function isn't
> currently configurable.  Which I think we've already accepted will be
> the case for some tables, until we go and make them configurable either
> by adding a table-specific option, adding a category, or both.
>
>> Since we can't distinguish these cases, then it makes more sense
>> when completion-category-overrides overrides everything:
>>
>>   (alist-get 'display-sort-function (alist-get category completion-category-overrides))
>>   (alist-get 'display-sort-function metadata) ;; metadata with/out individual options
>>   (alist-get 'display-sort-function (alist-get category completion-category-defaults))
>>
>> There is no problem with this because completion-category-overrides
>> is a user option as well, so everything still is under user control.
>
> Only if the completion table specifies a category.  Which most do not.
>
> So we'd need to change it to specify a category.

Agreed.

> And if we're doing that, we could also change it to not hard-code
> a display-sort-function at the same time.

Disagreed when this means to add an option in all cases.

> That is, for any tables where the display-sort-function is currently
> hardcoded, we can add a category, and remove the hardcoded
> display-sort-function from the table metadata, and add the
> display-sort-function to completion-category-defaults.
>
>>> So then we're only disagreeing over whether such options should exist?
>>
>> Yes, I think we should add individual options only in exceptional cases.
>>
>>> These individual options would also provide a natural place to document
>>> behavior like "if you configure the display-sort-function for buffer
>>> completion to 'identity, then the buffer sort order will match
>>> (buffer-list)".  But the user could still make use of that information
>>> by configuring the category.
>>
>> I agree that an option with documentation could help in such cases
>> when a non-trivial sorting function is provided for a completion table.
>>
>>>> I see no need to add individual options as all.  Every completion table
>>>> that significantly differs from other tables so that it needs a separate
>>>> display-sort-function, could provide a separate category.  For example,
>>>> there is a category 'buffer'.  If 'switch-to-buffer' needs another
>>>> display-sort-function it could provide a category 'buffer-for-switching'.
>>>
>>> That won't work with the scenario I described before with sorting
>>> file-name completion by mtime, where changing the sorting requires also
>>> changing the completion table.
>>
>> I agree that individual options are required in such rare cases when
>> their values affect the completion table and its formatting.
>
> OK, I think I can agree with that, if we agree that in those rare cases,
> the individual options should take precedence over the category-based
> configuration.

The individual options can't take precedence until all tables don't
hardcode metadata that is hardly achievable.

>>> Also, this would require adding a category for essentially every
>>> completion table.  For example, I see that read-from-kill-ring specifies
>>> a display-sort-function, currently set to 'identity.
>>
>> It's much simpler to add an extra line with a category.
>>
>>> If we wanted to make that configurable, it seems much easier to just do
>>>
>>>           (if (eq action 'metadata)
>>>               ;; Keep sorted by recency
>>> -             '(metadata (display-sort-function . identity))
>>> +             `(metadata (display-sort-function . ,read-from-kill-ring-sort))
>>>             (complete-with-action action completions string pred)))
>>
>> This is an incomplete patch, there should be also a dozen of lines
>> with defcustom, its docstring, the version number and a list
>> of choices, etc.  And all this for a very small percent of users
>> who would like to change this order.  This is too wasteful.
>> It would be much more efficient to allow doing the same
>> by customizing completion-category-overrides.
>
> The docstring and list of choices for read-from-kill-ring-sort are
> something we want anyway - we would like to document that 'identity for
> read-for-kill-ring keeps the kill ring sorted by recency, for example.
> I see no better place to document that.
>
> The version number is also something we want anyway: if we just add a
> category to read-from-kill-ring in Emacs 30, this will work only in
> Emacs 30 and not in Emacs 29, and there's no way for a user to know that
> other than by reading NEWS.

I don't think such unnecessary defcustoms should be added lightly,
even documentation is of no help for such obvious things as 'identity'
that intuitively is understandable as keeping the original order.

> For such tables, I see three good possibilities (in order of my own
> preference):
>
> A.
> - Add read-from-kill-ring-sort defaulting to identity (with docstring)
> (diff is 1 line + defcustom)
>
> B.
> - Add read-from-kill-ring-sort defaulting to nil (with docstring)
> - add the 'read-kill category to the metadata
> - add 'read-kill to completion-category-defaults
> (diff is 3 lines + defcustom)
>
> C.
> - Remove display-sort-function from the metadata
> - add the 'read-kill category to the metadata
> - add 'read-kill to completion-category-defaults
> (diff is 3 lines)
>
> If you really don't want the defcustom and associated documentation, I'm
> OK with C.
>
> The option which I think is not good is:
>
> D.
> - add the 'read-kill category to the metadata
> - make completion-category-overrides take precedence over what is
>   specified in the table metadata
>
> (diff is 1 line)
>
> This is a slightly smaller diff than option C, but I think it's a
> fundamentally worse approach than C, because in the rare cases where we
> do want an individual option for the table, we won't have a way for that
> option to take precedence over the broader category-based configuration.

I still don't understand why do you worry about this precedence when
the user option completion-category-overrides is nil by default.

Could you describe a use cases when such precedence might become a problem?

Re: Updating *Completions* as you type

[sbaugh]

Juri Linkov <juri@linkov.net> writes:
>>>>>> - Again, the user is still able to configure the display-sort-function
>>>>>>   by configuring the individual completion table.
>>>>>
>>>>> Does this mean that every individual completion table should have
>>>>> a separate user option?
>>>>
>>>> No: only the completion tables which specify a display-sort-function in
>>>> their metadata.  All such completion tables should have a user option to
>>>> configure that display-sort-function.
>>>
>>> How then users could change the sorting order for individual tables
>>> that don't specify a display-sort-function to use an order different
>>> from completions-sort?
>>
>> They can use the category if the table specifies one.
>>
>> If the table neither specifies a category nor provides a table-specific
>> option, the display sort function for that table isn't currently
>> configurable.  Which I think we're both fine with?
>
> I think we should gradually add a category to most completion tables
> to make them completely configurable, not just with display-sort-function,
> but with all possible metadata.
>
> Adding a category resembles a long-lasting and still ongoing process of
> adding specific minibuffer history as a symbol to the HIST argument
> of different calls of read-from-minibuffer.

Agreed.

>>> The problem is that we can't distinguish two cases:
>>>
>>> 1. when display-sort-function is hard-coded in metadata
>>>    by the author of the completion table;
>>> 2. when display-sort-function in metadata
>>>    gets the value from the user option.
>>
>> I think we should just eliminate any instances of case 1.
>
> I don't think this is realistic to add an individual option in all cases.

That's not necessary.  We could also do possibility C that I described
before:

>> C.
>> - Remove display-sort-function from the metadata
>> - add the 'read-kill category to the metadata
>> - add 'read-kill to completion-category-defaults
>> (diff is 3 lines)

That seems simple and straightforward.

>>>> If we wanted to make that configurable, it seems much easier to just do
>>>>
>>>>           (if (eq action 'metadata)
>>>>               ;; Keep sorted by recency
>>>> -             '(metadata (display-sort-function . identity))
>>>> +             `(metadata (display-sort-function . ,read-from-kill-ring-sort))
>>>>             (complete-with-action action completions string pred)))
>>>
>>> This is an incomplete patch, there should be also a dozen of lines
>>> with defcustom, its docstring, the version number and a list
>>> of choices, etc.  And all this for a very small percent of users
>>> who would like to change this order.  This is too wasteful.
>>> It would be much more efficient to allow doing the same
>>> by customizing completion-category-overrides.
>>
>> The docstring and list of choices for read-from-kill-ring-sort are
>> something we want anyway - we would like to document that 'identity for
>> read-for-kill-ring keeps the kill ring sorted by recency, for example.
>> I see no better place to document that.
>>
>> The version number is also something we want anyway: if we just add a
>> category to read-from-kill-ring in Emacs 30, this will work only in
>> Emacs 30 and not in Emacs 29, and there's no way for a user to know that
>> other than by reading NEWS.
>
> I don't think such unnecessary defcustoms should be added lightly,
> even documentation is of no help for such obvious things as 'identity'
> that intuitively is understandable as keeping the original order.

Identity obviously keeps the original order, but what is the original
order?  That is not documented anywhere and I don't think it's
intuitive.  The user can always just try it and see, but that's a poor
substitute for documentation.

>> For such tables, I see three good possibilities (in order of my own
>> preference):
>>
>> A.
>> - Add read-from-kill-ring-sort defaulting to identity (with docstring)
>> (diff is 1 line + defcustom)
>>
>> B.
>> - Add read-from-kill-ring-sort defaulting to nil (with docstring)
>> - add the 'read-kill category to the metadata
>> - add 'read-kill to completion-category-defaults
>> (diff is 3 lines + defcustom)
>>
>> C.
>> - Remove display-sort-function from the metadata
>> - add the 'read-kill category to the metadata
>> - add 'read-kill to completion-category-defaults
>> (diff is 3 lines)
>>
>> If you really don't want the defcustom and associated documentation, I'm
>> OK with C.
>>
>> The option which I think is not good is:
>>
>> D.
>> - add the 'read-kill category to the metadata
>> - make completion-category-overrides take precedence over what is
>>   specified in the table metadata
>>
>> (diff is 1 line)
>>
>> This is a slightly smaller diff than option C, but I think it's a
>> fundamentally worse approach than C, because in the rare cases where we
>> do want an individual option for the table, we won't have a way for that
>> option to take precedence over the broader category-based configuration.
>
> I still don't understand why do you worry about this precedence when
> the user option completion-category-overrides is nil by default.
>
> Could you describe a use cases when such precedence might become a problem?

If some table needs an individual option (because the sorting needs to
affect the completion generation), but the table shares a category with
other tables, then that individual option will be overridden by the
category configuration.

For example, project-prompt-project-name allows one to complete over
project names.  If I wanted to sort its completions by some detail of
the underlying project (how recently the git repo was updated, maybe),
that would require the table to change behavior.  So it would need an
individual option.

However, project-prompt-project-name uses the same category as
project-find-file.  So if the user configured sorting for
project-find-file, it will override the table-specific option for
project-prompt-project-name.

I suppose another option is to simply declare that every table has to
have a unique category.  That would make "category" a misnomer though.

Re: Updating *Completions* as you type

[Juri Linkov]

>> I don't think this is realistic to add an individual option in all cases.
>
> That's not necessary.  We could also do possibility C that I described
> before:
>
>>> C.
>>> - Remove display-sort-function from the metadata
>>> - add the 'read-kill category to the metadata
>>> - add 'read-kill to completion-category-defaults
>>> (diff is 3 lines)
>
> That seems simple and straightforward.

Removing display-sort-function is still less safe
than just adding a category.

> Identity obviously keeps the original order, but what is the original
> order?  That is not documented anywhere and I don't think it's
> intuitive.  The user can always just try it and see, but that's a poor
> substitute for documentation.

In these rare cases when the default order is not intuitive,
this can be explained in the docstring of the command that uses
'completing-read', e.g. in the docstring of 'read-from-kill-ring'.

>> I still don't understand why do you worry about this precedence when
>> the user option completion-category-overrides is nil by default.
>>
>> Could you describe a use cases when such precedence might become a problem?
>
> If some table needs an individual option (because the sorting needs to
> affect the completion generation), but the table shares a category with
> other tables, then that individual option will be overridden by the
> category configuration.

Agreed, this is a problem.

> For example, project-prompt-project-name allows one to complete over
> project names.  If I wanted to sort its completions by some detail of
> the underlying project (how recently the git repo was updated, maybe),
> that would require the table to change behavior.  So it would need an
> individual option.

Or an individual subcategory.

> However, project-prompt-project-name uses the same category as
> project-find-file.  So if the user configured sorting for
> project-find-file, it will override the table-specific option for
> project-prompt-project-name.

I believe they should use two different subcategories, e.g.
'project-file' and 'project-name'.

> I suppose another option is to simply declare that every table has to
> have a unique category.  That would make "category" a misnomer though.

Even such subcategories as 'project-name' make sense to use in other
possible cases when reading a project name.

Re: Updating *Completions* as you type

[Spencer Baugh]

Juri Linkov <juri@linkov.net> writes:

>>> I don't think this is realistic to add an individual option in all cases.
>>
>> That's not necessary.  We could also do possibility C that I described
>> before:
>>
>>>> C.
>>>> - Remove display-sort-function from the metadata
>>>> - add the 'read-kill category to the metadata
>>>> - add 'read-kill to completion-category-defaults
>>>> (diff is 3 lines)
>>
>> That seems simple and straightforward.
>
> Removing display-sort-function is still less safe
> than just adding a category.

Why do you say that?

The reason that comes to mind is that there are replacement completion
UIs which will need to explicitly add support for the category.  So
removing display-sort-function will affect them immediately, when they
might not yet have support for getting display-sort-function from
completion-category-defaults.

That is true.

But that actually suggests a further argument in this direction: if we
use user options which change the display-sort-function in the table
metadata, we'll have support for all completion UIs out of the box.

That seems really desirable!  So maybe we do want a solution like A
where we add a user option?  Since that user option will work for all
completion UIs.

Announcing "you can now customize the sorting order of a bunch of
completing-read-based things in this new way" but having that new way
only work for the default completion UI is a bit sad, although of course
they can support the new way eventually.

>> Identity obviously keeps the original order, but what is the original
>> order?  That is not documented anywhere and I don't think it's
>> intuitive.  The user can always just try it and see, but that's a poor
>> substitute for documentation.
>
> In these rare cases when the default order is not intuitive,
> this can be explained in the docstring of the command that uses
> 'completing-read', e.g. in the docstring of 'read-from-kill-ring'.

Hm, I do think the wording on that would be a bit tricky.  Since
*actually* the default behavior today is alphabetical sorting.  We would
want to say "if completions-sort is nil, read-buffer completions are in
buffer-list order".

I guess it's not too bad, but we also need to document the category
symbol.  And perhaps the version it was added in.  All together it still
seems to me that it would be better to just mention
'read-from-kill-ring-sort' in the docstring of 'read-from-kill-ring'.

>>> I still don't understand why do you worry about this precedence when
>>> the user option completion-category-overrides is nil by default.
>>>
>>> Could you describe a use cases when such precedence might become a problem?
>>
>> If some table needs an individual option (because the sorting needs to
>> affect the completion generation), but the table shares a category with
>> other tables, then that individual option will be overridden by the
>> category configuration.
>
> Agreed, this is a problem.
>
>> For example, project-prompt-project-name allows one to complete over
>> project names.  If I wanted to sort its completions by some detail of
>> the underlying project (how recently the git repo was updated, maybe),
>> that would require the table to change behavior.  So it would need an
>> individual option.
>
> Or an individual subcategory.
>
>> However, project-prompt-project-name uses the same category as
>> project-find-file.  So if the user configured sorting for
>> project-find-file, it will override the table-specific option for
>> project-prompt-project-name.
>
> I believe they should use two different subcategories, e.g.
> 'project-file' and 'project-name'.

I agree, but...

>> I suppose another option is to simply declare that every table has to
>> have a unique category.  That would make "category" a misnomer though.
>
> Even such subcategories as 'project-name' make sense to use in other
> possible cases when reading a project name.

...if the project-name category is used for other tables too, but the
option is supposed to be specific to an individual completion table,
then we have the same problem again.

Re: Updating *Completions* as you type

[Juri Linkov]

>>>> I don't think this is realistic to add an individual option in all cases.
>>>
>>> That's not necessary.  We could also do possibility C that I described
>>> before:
>>>
>>>>> C.
>>>>> - Remove display-sort-function from the metadata
>>>>> - add the 'read-kill category to the metadata
>>>>> - add 'read-kill to completion-category-defaults
>>>>> (diff is 3 lines)
>>>
>>> That seems simple and straightforward.
>>
>> Removing display-sort-function is still less safe
>> than just adding a category.
>
> Why do you say that?
>
> The reason that comes to mind is that there are replacement completion
> UIs which will need to explicitly add support for the category.  So
> removing display-sort-function will affect them immediately, when they
> might not yet have support for getting display-sort-function from
> completion-category-defaults.
>
> That is true.
>
> But that actually suggests a further argument in this direction: if we
> use user options which change the display-sort-function in the table
> metadata, we'll have support for all completion UIs out of the box.
>
> That seems really desirable!  So maybe we do want a solution like A
> where we add a user option?  Since that user option will work for all
> completion UIs.
>
> Announcing "you can now customize the sorting order of a bunch of
> completing-read-based things in this new way" but having that new way
> only work for the default completion UI is a bit sad, although of course
> they can support the new way eventually.

This is what I believe they should do: we add a category,
and they support it as well.

>>> Identity obviously keeps the original order, but what is the original
>>> order?  That is not documented anywhere and I don't think it's
>>> intuitive.  The user can always just try it and see, but that's a poor
>>> substitute for documentation.
>>
>> In these rare cases when the default order is not intuitive,
>> this can be explained in the docstring of the command that uses
>> 'completing-read', e.g. in the docstring of 'read-from-kill-ring'.
>
> Hm, I do think the wording on that would be a bit tricky.  Since
> *actually* the default behavior today is alphabetical sorting.  We would
> want to say "if completions-sort is nil, read-buffer completions are in
> buffer-list order".
>
> I guess it's not too bad, but we also need to document the category
> symbol.  And perhaps the version it was added in.  All together it still
> seems to me that it would be better to just mention
> 'read-from-kill-ring-sort' in the docstring of 'read-from-kill-ring'.

I'm still not convinced that a user option is needed for such minor thing.

>>>> I still don't understand why do you worry about this precedence when
>>>> the user option completion-category-overrides is nil by default.
>>>>
>>>> Could you describe a use cases when such precedence might become a problem?
>>>
>>> If some table needs an individual option (because the sorting needs to
>>> affect the completion generation), but the table shares a category with
>>> other tables, then that individual option will be overridden by the
>>> category configuration.
>>
>> Agreed, this is a problem.
>>
>>> For example, project-prompt-project-name allows one to complete over
>>> project names.  If I wanted to sort its completions by some detail of
>>> the underlying project (how recently the git repo was updated, maybe),
>>> that would require the table to change behavior.  So it would need an
>>> individual option.
>>
>> Or an individual subcategory.
>>
>>> However, project-prompt-project-name uses the same category as
>>> project-find-file.  So if the user configured sorting for
>>> project-find-file, it will override the table-specific option for
>>> project-prompt-project-name.
>>
>> I believe they should use two different subcategories, e.g.
>> 'project-file' and 'project-name'.
>
> I agree, but...
>
>>> I suppose another option is to simply declare that every table has to
>>> have a unique category.  That would make "category" a misnomer though.
>>
>> Even such subcategories as 'project-name' make sense to use in other
>> possible cases when reading a project name.
>
> ...if the project-name category is used for other tables too, but the
> option is supposed to be specific to an individual completion table,
> then we have the same problem again.

And an alternative to add separate options to all these tables
doesn't look more attractive.

Re: Updating *Completions* as you type

[sbaugh]

[-- Attachment #1: Type: text/plain, Size: 4169 bytes --]

Juri Linkov <juri@linkov.net> writes:
>>>>> I don't think this is realistic to add an individual option in all cases.
>>>>
>>>> That's not necessary.  We could also do possibility C that I described
>>>> before:
>>>>
>>>>>> C.
>>>>>> - Remove display-sort-function from the metadata
>>>>>> - add the 'read-kill category to the metadata
>>>>>> - add 'read-kill to completion-category-defaults
>>>>>> (diff is 3 lines)
>>>>
>>>> That seems simple and straightforward.
>>>
>>> Removing display-sort-function is still less safe
>>> than just adding a category.
>>
>> Why do you say that?
>>
>> The reason that comes to mind is that there are replacement completion
>> UIs which will need to explicitly add support for the category.  So
>> removing display-sort-function will affect them immediately, when they
>> might not yet have support for getting display-sort-function from
>> completion-category-defaults.
>>
>> That is true.
>>
>> But that actually suggests a further argument in this direction: if we
>> use user options which change the display-sort-function in the table
>> metadata, we'll have support for all completion UIs out of the box.
>>
>> That seems really desirable!  So maybe we do want a solution like A
>> where we add a user option?  Since that user option will work for all
>> completion UIs.
>>
>> Announcing "you can now customize the sorting order of a bunch of
>> completing-read-based things in this new way" but having that new way
>> only work for the default completion UI is a bit sad, although of course
>> they can support the new way eventually.
>
> This is what I believe they should do: we add a category,
> and they support it as well.

OK, I'm fine with that, but when we do that, I think the per-table
option should override the per-category option.

>>>>> I still don't understand why do you worry about this precedence when
>>>>> the user option completion-category-overrides is nil by default.
>>>>>
>>>>> Could you describe a use cases when such precedence might become a problem?
>>>>
>>>> If some table needs an individual option (because the sorting needs to
>>>> affect the completion generation), but the table shares a category with
>>>> other tables, then that individual option will be overridden by the
>>>> category configuration.
>>>
>>> Agreed, this is a problem.
>>>
>>>> For example, project-prompt-project-name allows one to complete over
>>>> project names.  If I wanted to sort its completions by some detail of
>>>> the underlying project (how recently the git repo was updated, maybe),
>>>> that would require the table to change behavior.  So it would need an
>>>> individual option.
>>>
>>> Or an individual subcategory.
>>>
>>>> However, project-prompt-project-name uses the same category as
>>>> project-find-file.  So if the user configured sorting for
>>>> project-find-file, it will override the table-specific option for
>>>> project-prompt-project-name.
>>>
>>> I believe they should use two different subcategories, e.g.
>>> 'project-file' and 'project-name'.
>>
>> I agree, but...
>>
>>>> I suppose another option is to simply declare that every table has to
>>>> have a unique category.  That would make "category" a misnomer though.
>>>
>>> Even such subcategories as 'project-name' make sense to use in other
>>> possible cases when reading a project name.
>>
>> ...if the project-name category is used for other tables too, but the
>> option is supposed to be specific to an individual completion table,
>> then we have the same problem again.
>
> And an alternative to add separate options to all these tables
> doesn't look more attractive.

Yes, but we don't have to do that, I'm OK with a category-based
approach.  I just think we should reserve the *ability* to use
table-specific options, by making a table-specific display-sort-function
override the category-specific display-sort-function.

Anyway, we're going around in circles a bit here.  How about this patch
which only adds the new historical option to completions-sort?  I think
we're in agreement on everything in this patch, and maybe installing it
will get some user feedback which we can use when coming back to this
later.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Add-historical-option-to-completions-sort.patch --]
[-- Type: text/x-patch, Size: 5242 bytes --]

From c4435be3689278d380099f2341ac242bf74639df Mon Sep 17 00:00:00 2001
From: Spencer Baugh <sbaugh@catern.com>
Date: Tue, 17 Oct 2023 09:09:55 -0400
Subject: [PATCH] Add historical option to completions-sort

Support sorting candidates in *Completions* by the order they show up
in the minibuffer history.

Also add minibuffer-sort-alphabetically and
minibuffer-sort-by-history, which are usable for both completions-sort
and display-sort-function.

* lisp/minibuffer.el (completions-sort): Document 'historical option.
(minibuffer-completion-help): Support 'historical option.
(minibuffer-sort-alphabetically)
(minibuffer-completion-base, minibuffer-sort-by-history): Add.
---
 lisp/minibuffer.el | 63 +++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 57 insertions(+), 6 deletions(-)

diff --git a/lisp/minibuffer.el b/lisp/minibuffer.el
index 5c12d9fc914..dfb30e4364c 100644
--- a/lisp/minibuffer.el
+++ b/lisp/minibuffer.el
@@ -1314,14 +1314,26 @@ completion-cycle-threshold
 (defcustom completions-sort 'alphabetical
   "Sort candidates in the *Completions* buffer.
 
-The value can be nil to disable sorting, `alphabetical' for
-alphabetical sorting or a custom sorting function.  The sorting
-function takes and returns a list of completion candidate
-strings."
+Completion candidates in the *Completions* buffer are sorted
+depending on the value.
+
+If nil, sorting is disabled.
+If `alphabetical', candidates are sorted by
+`minibuffer-sort-alphabetically'.
+If `historical', candidates are sorted by
+`minibuffer-sort-by-history'.
+If a function, the function is called to sort the candidates.
+The sorting function takes and returns a list of completion
+candidate strings.
+
+If the completion-specific metadata provides a
+`display-sort-function', that is used instead and this value is
+ignored."
   :type '(choice (const :tag "No sorting" nil)
                  (const :tag "Alphabetical sorting" alphabetical)
+                 (const :tag "Historical sorting" historical)
                  (function :tag "Custom function"))
-  :version "29.1")
+  :version "30.1")
 
 (defcustom completions-group nil
   "Enable grouping of completion candidates in the *Completions* buffer.
@@ -1647,6 +1659,43 @@ minibuffer--sort-preprocess-history
                      (substring c base-size)))
                  hist)))))
 
+(defun minibuffer-sort-alphabetically (completions)
+  "Sort COMPLETIONS alphabetically.
+
+COMPLETIONS are sorted alphabetically by `string-lessp'.
+
+This is a suitable function to use for `completions-sort' or to
+include as `display-sort-function' in completion metadata."
+  (sort completions #'string-lessp))
+
+(defvar minibuffer-completion-base nil
+  "The base for the current completion.
+
+This is the part of the current minibuffer input which is not
+being completed on.  This is primarily relevant for file names,
+where this is the directory component of the file name.")
+
+(defun minibuffer-sort-by-history (completions)
+  "Sort COMPLETIONS by their position in `minibuffer-history-variable'.
+
+COMPLETIONS are sorted first by `minibuffer-sort-alphbetically',
+then any elements occuring in the minibuffer history list are
+moved to the front based on the order they occur in the history.
+If a history variable hasn't been specified for this call of
+`completing-read', COMPLETIONS are sorted only by
+`minibuffer-sort-alphbetically'.
+
+This is a suitable function to use for `completions-sort' or to
+include as `display-sort-function' in completion metadata."
+  (let ((alphabetized (sort completions #'string-lessp)))
+    ;; Only use history when it's specific to these completions.
+    (if (eq minibuffer-history-variable
+            (default-value minibuffer-history-variable))
+        alphabetized
+      (minibuffer--sort-by-position
+       (minibuffer--sort-preprocess-history minibuffer-completion-base)
+       alphabetized))))
+
 (defun minibuffer--group-by (group-fun sort-fun elems)
   "Group ELEMS by GROUP-FUN and sort groups by SORT-FUN."
   (let ((groups))
@@ -2409,6 +2458,7 @@ minibuffer-completion-help
       (let* ((last (last completions))
              (base-size (or (cdr last) 0))
              (prefix (unless (zerop base-size) (substring string 0 base-size)))
+             (minibuffer-completion-base (substring string 0 base-size))
              (base-prefix (buffer-substring (minibuffer--completion-prompt-end)
                                             (+ start base-size)))
              (base-suffix
@@ -2473,7 +2523,8 @@ minibuffer-completion-help
                                             (funcall sort-fun completions)
                                           (pcase completions-sort
                                             ('nil completions)
-                                            ('alphabetical (sort completions #'string-lessp))
+                                            ('alphabetical (minibuffer-sort-alphabetically completions))
+                                            ('historical (minibuffer-sort-by-history completions))
                                             (_ (funcall completions-sort completions)))))
 
                       ;; After sorting, group the candidates using the
-- 
2.42.1

Re: Updating *Completions* as you type

[Juri Linkov]

>> This is what I believe they should do: we add a category,
>> and they support it as well.
>
> OK, I'm fine with that, but when we do that, I think the per-table
> option should override the per-category option.

I agree that the per-table option should override the per-category option,
but see no way to distinguish customized values from hard-coded ones
without trying to turn all hard-coded values into options.

display-buffer has a similar problem, but the difference is that
it's possible to identify a buffer by its name and use a regexp
to match buffer names.  Whereas for completing-read it's not easy
to identify a completion table.  A category matches a set of
completion tables, so maybe we need another identification
for individual tables?

> I'm OK with a category-based approach.  I just think we should reserve
> the *ability* to use table-specific options, by making a table-specific
> display-sort-function override the category-specific display-sort-function.

Probably introducing a new field to metadata could help to resolve ambiguities.

> Anyway, we're going around in circles a bit here.  How about this patch
> which only adds the new historical option to completions-sort?  I think
> we're in agreement on everything in this patch, and maybe installing it
> will get some user feedback which we can use when coming back to this
> later.

Thanks, everything looks nice, only a etc/NEWS announcement is missing.

Re: Updating *Completions* as you type

[Spencer Baugh]

[-- Attachment #1: Type: text/plain, Size: 1926 bytes --]

Juri Linkov <juri@linkov.net> writes:
>>> This is what I believe they should do: we add a category,
>>> and they support it as well.
>>
>> OK, I'm fine with that, but when we do that, I think the per-table
>> option should override the per-category option.
>
> I agree that the per-table option should override the per-category option,
> but see no way to distinguish customized values from hard-coded ones
> without trying to turn all hard-coded values into options.

Right.  I guess my position is that there's not that many hard-coded
display-sort-functions (only 5 in core Emacs), so turning them all to
options is fine.

> display-buffer has a similar problem, but the difference is that
> it's possible to identify a buffer by its name and use a regexp
> to match buffer names.  Whereas for completing-read it's not easy
> to identify a completion table.  A category matches a set of
> completion tables, so maybe we need another identification
> for individual tables?

True, that would help.  Maybe the function symbol for the completion
table?  Tables are usually lambdas today, but maybe we could make it
easy to use a defun'd function instead, which would be very good for
comprehensibility in general IMO.

>> I'm OK with a category-based approach.  I just think we should reserve
>> the *ability* to use table-specific options, by making a table-specific
>> display-sort-function override the category-specific display-sort-function.
>
> Probably introducing a new field to metadata could help to resolve ambiguities.
>
>> Anyway, we're going around in circles a bit here.  How about this patch
>> which only adds the new historical option to completions-sort?  I think
>> we're in agreement on everything in this patch, and maybe installing it
>> will get some user feedback which we can use when coming back to this
>> later.
>
> Thanks, everything looks nice, only a etc/NEWS announcement is missing.

Added.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Add-historical-option-to-completions-sort.patch --]
[-- Type: text/x-patch, Size: 5875 bytes --]

From 4196334888fb35395deca9418203f3622dedbaff Mon Sep 17 00:00:00 2001
From: Spencer Baugh <sbaugh@catern.com>
Date: Tue, 17 Oct 2023 09:09:55 -0400
Subject: [PATCH] Add historical option to completions-sort

Support sorting candidates in *Completions* by the order they show up
in the minibuffer history.

Also add minibuffer-sort-alphabetically and
minibuffer-sort-by-history, which are usable for both completions-sort
and display-sort-function.

* lisp/minibuffer.el (completions-sort): Document 'historical option.
(minibuffer-completion-help): Support 'historical option.
(minibuffer-sort-alphabetically)
(minibuffer-completion-base, minibuffer-sort-by-history): Add.
* etc/NEWS: Announce it.
---
 etc/NEWS           |  5 ++++
 lisp/minibuffer.el | 63 +++++++++++++++++++++++++++++++++++++++++-----
 2 files changed, 62 insertions(+), 6 deletions(-)

diff --git a/etc/NEWS b/etc/NEWS
index fd633fad6fb..eefa9692e07 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -620,6 +620,11 @@ completions window.  When the completions window is not visible,
 then all these keys have their usual meaning in the minibuffer.
 This option is supported for in-buffer completion as well.
 
+*** New value for 'historical' for user option 'completions-sort'
+When 'completions-sort' is set to 'historical', completion candidates
+will be sorted by their position in the minibuffer history, more
+recent candidates appearing first.
+
 ** Pcomplete
 
 ---
diff --git a/lisp/minibuffer.el b/lisp/minibuffer.el
index 5c12d9fc914..dfb30e4364c 100644
--- a/lisp/minibuffer.el
+++ b/lisp/minibuffer.el
@@ -1314,14 +1314,26 @@ completion-cycle-threshold
 (defcustom completions-sort 'alphabetical
   "Sort candidates in the *Completions* buffer.
 
-The value can be nil to disable sorting, `alphabetical' for
-alphabetical sorting or a custom sorting function.  The sorting
-function takes and returns a list of completion candidate
-strings."
+Completion candidates in the *Completions* buffer are sorted
+depending on the value.
+
+If nil, sorting is disabled.
+If `alphabetical', candidates are sorted by
+`minibuffer-sort-alphabetically'.
+If `historical', candidates are sorted by
+`minibuffer-sort-by-history'.
+If a function, the function is called to sort the candidates.
+The sorting function takes and returns a list of completion
+candidate strings.
+
+If the completion-specific metadata provides a
+`display-sort-function', that is used instead and this value is
+ignored."
   :type '(choice (const :tag "No sorting" nil)
                  (const :tag "Alphabetical sorting" alphabetical)
+                 (const :tag "Historical sorting" historical)
                  (function :tag "Custom function"))
-  :version "29.1")
+  :version "30.1")
 
 (defcustom completions-group nil
   "Enable grouping of completion candidates in the *Completions* buffer.
@@ -1647,6 +1659,43 @@ minibuffer--sort-preprocess-history
                      (substring c base-size)))
                  hist)))))
 
+(defun minibuffer-sort-alphabetically (completions)
+  "Sort COMPLETIONS alphabetically.
+
+COMPLETIONS are sorted alphabetically by `string-lessp'.
+
+This is a suitable function to use for `completions-sort' or to
+include as `display-sort-function' in completion metadata."
+  (sort completions #'string-lessp))
+
+(defvar minibuffer-completion-base nil
+  "The base for the current completion.
+
+This is the part of the current minibuffer input which is not
+being completed on.  This is primarily relevant for file names,
+where this is the directory component of the file name.")
+
+(defun minibuffer-sort-by-history (completions)
+  "Sort COMPLETIONS by their position in `minibuffer-history-variable'.
+
+COMPLETIONS are sorted first by `minibuffer-sort-alphbetically',
+then any elements occuring in the minibuffer history list are
+moved to the front based on the order they occur in the history.
+If a history variable hasn't been specified for this call of
+`completing-read', COMPLETIONS are sorted only by
+`minibuffer-sort-alphbetically'.
+
+This is a suitable function to use for `completions-sort' or to
+include as `display-sort-function' in completion metadata."
+  (let ((alphabetized (sort completions #'string-lessp)))
+    ;; Only use history when it's specific to these completions.
+    (if (eq minibuffer-history-variable
+            (default-value minibuffer-history-variable))
+        alphabetized
+      (minibuffer--sort-by-position
+       (minibuffer--sort-preprocess-history minibuffer-completion-base)
+       alphabetized))))
+
 (defun minibuffer--group-by (group-fun sort-fun elems)
   "Group ELEMS by GROUP-FUN and sort groups by SORT-FUN."
   (let ((groups))
@@ -2409,6 +2458,7 @@ minibuffer-completion-help
       (let* ((last (last completions))
              (base-size (or (cdr last) 0))
              (prefix (unless (zerop base-size) (substring string 0 base-size)))
+             (minibuffer-completion-base (substring string 0 base-size))
              (base-prefix (buffer-substring (minibuffer--completion-prompt-end)
                                             (+ start base-size)))
              (base-suffix
@@ -2473,7 +2523,8 @@ minibuffer-completion-help
                                             (funcall sort-fun completions)
                                           (pcase completions-sort
                                             ('nil completions)
-                                            ('alphabetical (sort completions #'string-lessp))
+                                            ('alphabetical (minibuffer-sort-alphabetically completions))
+                                            ('historical (minibuffer-sort-by-history completions))
                                             (_ (funcall completions-sort completions)))))
 
                       ;; After sorting, group the candidates using the
-- 
2.42.1

Re: Updating *Completions* as you type

[Eli Zaretskii]

> From: Spencer Baugh <sbaugh@catern.com>
> Date: Tue, 28 Nov 2023 14:38:55 +0000 (UTC)
> Cc: Spencer Baugh <sbaugh@janestreet.com>, emacs-devel@gnu.org
> 
> +*** New value for 'historical' for user option 'completions-sort'
                 ^^^
That "for" should be removed.

> +When 'completions-sort' is set to 'historical', completion candidates
> +will be sorted by their position in the minibuffer history, more
                           ^^^^^^^^
Not "position", but chronological order.

> +Completion candidates in the *Completions* buffer are sorted
> +depending on the value.
> +
> +If nil, sorting is disabled.

"If it's nil, sorting is disabled."

> +If `alphabetical', candidates are sorted by
> +`minibuffer-sort-alphabetically'.

"If it's `alphabetical', candidates are sorted..."

Etc.

Also, did you make sure these symbols are not highlighted as links
even if a function or variable by that name exists?  I think you
should use this technique from "Documentation Tips" to prevent that:

     If a symbol has a function definition and/or a variable definition,
     but those are irrelevant to the use of the symbol that you are
     documenting, you can write the words ‘symbol’ or ‘program’ before
     the symbol name to prevent making any hyperlink.

> +If a function, the function is called to sort the candidates.
> +The sorting function takes and returns a list of completion
> +candidate strings.

Can it return the same list, or should it always return a copy?

> +If the completion-specific metadata provides a
> +`display-sort-function', that is used instead and this value is
> +ignored."

"If the completion-specific metadata provides
a`display-sort-function', that function overrides the value of this
variable."

>    :type '(choice (const :tag "No sorting" nil)
>                   (const :tag "Alphabetical sorting" alphabetical)
> +                 (const :tag "Historical sorting" historical)
                                 ^^^^^^^^^^^^^^^^^^
"Chronological sorting"

> +(defvar minibuffer-completion-base nil
> +  "The base for the current completion.
> +
> +This is the part of the current minibuffer input which is not
> +being completed on.

I couldn't understand what this sentence means.

> +(defun minibuffer-sort-by-history (completions)
> +  "Sort COMPLETIONS by their position in `minibuffer-history-variable'.
> +
> +COMPLETIONS are sorted first by `minibuffer-sort-alphbetically',
> +then any elements occuring in the minibuffer history list are
> +moved to the front based on the order they occur in the history.
> +If a history variable hasn't been specified for this call of
> +`completing-read', COMPLETIONS are sorted only by
> +`minibuffer-sort-alphbetically'.

Again, please use "chronologically" here, not "by history".

Re: Updating *Completions* as you type

[Juri Linkov]

>>    :type '(choice (const :tag "No sorting" nil)
>>                   (const :tag "Alphabetical sorting" alphabetical)
>> +                 (const :tag "Historical sorting" historical)
>                                  ^^^^^^^^^^^^^^^^^^
> "Chronological sorting"

Chronological sorting is how a list of buffers sorted by recency.
Historical sorting is how items in history are sorted by input.

Re: Updating *Completions* as you type

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: Spencer Baugh <sbaugh@catern.com>,  sbaugh@janestreet.com,
>   emacs-devel@gnu.org
> Date: Tue, 28 Nov 2023 19:13:01 +0200
> 
> >>    :type '(choice (const :tag "No sorting" nil)
> >>                   (const :tag "Alphabetical sorting" alphabetical)
> >> +                 (const :tag "Historical sorting" historical)
> >                                  ^^^^^^^^^^^^^^^^^^
> > "Chronological sorting"
> 
> Chronological sorting is how a list of buffers sorted by recency.
> Historical sorting is how items in history are sorted by input.

Aren't they the same order?  If not, why not?

Re: Updating *Completions* as you type

[Juri Linkov]

>> >>    :type '(choice (const :tag "No sorting" nil)
>> >>                   (const :tag "Alphabetical sorting" alphabetical)
>> >> +                 (const :tag "Historical sorting" historical)
>> >                                  ^^^^^^^^^^^^^^^^^^
>> > "Chronological sorting"
>>
>> Chronological sorting is how a list of buffers sorted by recency.
>> Historical sorting is how items in history are sorted by input.
>
> Aren't they the same order?  If not, why not?

"Chronological" is more wide and thus more ambiguous and confusing for users.

Re: Updating *Completions* as you type

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: sbaugh@catern.com,  sbaugh@janestreet.com,  emacs-devel@gnu.org
> Date: Wed, 29 Nov 2023 09:11:24 +0200
> 
> >> >>    :type '(choice (const :tag "No sorting" nil)
> >> >>                   (const :tag "Alphabetical sorting" alphabetical)
> >> >> +                 (const :tag "Historical sorting" historical)
> >> >                                  ^^^^^^^^^^^^^^^^^^
> >> > "Chronological sorting"
> >>
> >> Chronological sorting is how a list of buffers sorted by recency.
> >> Historical sorting is how items in history are sorted by input.
> >
> > Aren't they the same order?  If not, why not?
> 
> "Chronological" is more wide and thus more ambiguous and confusing for users.

Sorry, I still don't think I follow.  Could you perhaps elaborate
about how "chronological" is more wide?

My problem with "history order" or "historical order" is that it could
be confusing, since we are talking about input history.  So we in fact
saying something like "sort history in history order".  I'd like to
avoid using the same word twice in two different meanings, as that is
bound to confuse someone.

Re: Updating *Completions* as you type

[Spencer Baugh]

[-- Attachment #1: Type: text/plain, Size: 1607 bytes --]

On Wed, Nov 29, 2023, 8:10 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Juri Linkov <juri@linkov.net>
> > Cc: sbaugh@catern.com,  sbaugh@janestreet.com,  emacs-devel@gnu.org
> > Date: Wed, 29 Nov 2023 09:11:24 +0200
> >
> > >> >>    :type '(choice (const :tag "No sorting" nil)
> > >> >>                   (const :tag "Alphabetical sorting" alphabetical)
> > >> >> +                 (const :tag "Historical sorting" historical)
> > >> >                                  ^^^^^^^^^^^^^^^^^^
> > >> > "Chronological sorting"
> > >>
> > >> Chronological sorting is how a list of buffers sorted by recency.
> > >> Historical sorting is how items in history are sorted by input.
> > >
> > > Aren't they the same order?  If not, why not?
> >
> > "Chronological" is more wide and thus more ambiguous and confusing for
> users.
>
> Sorry, I still don't think I follow.  Could you perhaps elaborate
> about how "chronological" is more wide?
>
> My problem with "history order" or "historical order" is that it could
> be confusing, since we are talking about input history.  So we in fact
> saying something like "sort history in history order".


Oh, no, this is about sorting completion candidates, not history. The
history is still in chronological order always, it's not affected by this
setting. The setting just determines whether completion candidates are
sorted alphabetically or to match the history.

So it's "sort completions in history order".

I'd like to
> avoid using the same word twice in two different meanings, as that is
> bound to confuse someone.
>

[-- Attachment #2: Type: text/html, Size: 2974 bytes --]

Re: Updating *Completions* as you type

[Eli Zaretskii]

> From: Spencer Baugh <sbaugh@janestreet.com>
> Date: Wed, 29 Nov 2023 09:14:05 -0500
> Cc: Juri Linkov <juri@linkov.net>, Spencer Baugh <sbaugh@catern.com>, emacs-devel@gnu.org
> 
>  My problem with "history order" or "historical order" is that it could
>  be confusing, since we are talking about input history.  So we in fact
>  saying something like "sort history in history order".
> 
> Oh, no, this is about sorting completion candidates, not history. The history is still in chronological
> order always, it's not affected by this setting. The setting just determines whether completion
> candidates are sorted alphabetically or to match the history.
> 
> So it's "sort completions in history order".

"History order" still sounds too heavy-handed.  How about "in the
order of their typing" or somesuch?

Re: Updating *Completions* as you type

[Spencer Baugh]

[-- Attachment #1: Type: text/plain, Size: 1123 bytes --]

On Wed, Nov 29, 2023 at 9:54 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Spencer Baugh <sbaugh@janestreet.com>
> > Date: Wed, 29 Nov 2023 09:14:05 -0500
> > Cc: Juri Linkov <juri@linkov.net>, Spencer Baugh <sbaugh@catern.com>,
> emacs-devel@gnu.org
> >
> >  My problem with "history order" or "historical order" is that it could
> >  be confusing, since we are talking about input history.  So we in fact
> >  saying something like "sort history in history order".
> >
> > Oh, no, this is about sorting completion candidates, not history. The
> history is still in chronological
> > order always, it's not affected by this setting. The setting just
> determines whether completion
> > candidates are sorted alphabetically or to match the history.
> >
> > So it's "sort completions in history order".
>
> "History order" still sounds too heavy-handed.  How about "in the
> order of their typing" or somesuch?
>

Why heavy-handed?  I feel confident that more users will understand
"History order" than "in the order of their typing".

How about "Sorted in the order of minibuffer history"?

[-- Attachment #2: Type: text/html, Size: 1822 bytes --]

Re: Updating *Completions* as you type

[Eli Zaretskii]

> From: Spencer Baugh <sbaugh@janestreet.com>
> Date: Wed, 29 Nov 2023 10:21:47 -0500
> Cc: juri@linkov.net, sbaugh@catern.com, emacs-devel@gnu.org
> 
>  > So it's "sort completions in history order".
> 
>  "History order" still sounds too heavy-handed.  How about "in the
>  order of their typing" or somesuch?
> 
> Why heavy-handed?

Because "history" has meaning outside of Emacs.

> I feel confident that more users will understand "History order" than "in the
> order of their typing".
> 
> How about "Sorted in the order of minibuffer history"?

That's better, but if you are okay with that, how about "Sorted in the
chronological order of minibuffer input"?

Re: Updating *Completions* as you type

[Spencer Baugh]

[-- Attachment #1: Type: text/plain, Size: 720 bytes --]

On Wed, Nov 29, 2023 at 10:52 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Spencer Baugh <sbaugh@janestreet.com>
> > Date: Wed, 29 Nov 2023 10:21:47 -0500
> > Cc: juri@linkov.net, sbaugh@catern.com, emacs-devel@gnu.org
> > I feel confident that more users will understand "History order" than
> "in the
> > order of their typing".
> >
> > How about "Sorted in the order of minibuffer history"?
>
> That's better, but if you are okay with that, how about "Sorted in the
> chronological order of minibuffer input"?
>

That does not mention history, so I think it's worse - how are user
supposed to know it's about history?  But I'm okay with "Sorted in
chronological order of minibuffer history".

[-- Attachment #2: Type: text/html, Size: 1321 bytes --]

Re: Updating *Completions* as you type

[Eli Zaretskii]

> From: Spencer Baugh <sbaugh@janestreet.com>
> Date: Wed, 29 Nov 2023 14:17:57 -0500
> Cc: juri@linkov.net, sbaugh@catern.com, emacs-devel@gnu.org
> 
> On Wed, Nov 29, 2023 at 10:52 AM Eli Zaretskii <eliz@gnu.org> wrote:
> 
>  > From: Spencer Baugh <sbaugh@janestreet.com>
>  > Date: Wed, 29 Nov 2023 10:21:47 -0500
>  > Cc: juri@linkov.net, sbaugh@catern.com, emacs-devel@gnu.org
>  > I feel confident that more users will understand "History order" than "in the
>  > order of their typing".
>  > 
>  > How about "Sorted in the order of minibuffer history"?
> 
>  That's better, but if you are okay with that, how about "Sorted in the
>  chronological order of minibuffer input"?
> 
> That does not mention history, so I think it's worse - how are user supposed to know it's about
> history?

?? Didn't you yourself say several messages ago it was NOT about
history?  I quote:

> > Oh, no, this is about sorting completion candidates, not history.

So now you say this IS about history?

> But I'm okay with "Sorted in chronological order of minibuffer history".

This is tautology: history is always in chronological order.  We are
in effect wasting one word by repeating another.

Re: Updating *Completions* as you type

[Spencer Baugh]

[-- Attachment #1: Type: text/plain, Size: 1691 bytes --]

On Thu, Nov 30, 2023, 1:12 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Spencer Baugh <sbaugh@janestreet.com>
> > Date: Wed, 29 Nov 2023 14:17:57 -0500
> > Cc: juri@linkov.net, sbaugh@catern.com, emacs-devel@gnu.org
> >
> > On Wed, Nov 29, 2023 at 10:52 AM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> >  > From: Spencer Baugh <sbaugh@janestreet.com>
> >  > Date: Wed, 29 Nov 2023 10:21:47 -0500
> >  > Cc: juri@linkov.net, sbaugh@catern.com, emacs-devel@gnu.org
> >  > I feel confident that more users will understand "History order" than
> "in the
> >  > order of their typing".
> >  >
> >  > How about "Sorted in the order of minibuffer history"?
> >
> >  That's better, but if you are okay with that, how about "Sorted in the
> >  chronological order of minibuffer input"?
> >
> > That does not mention history, so I think it's worse - how are user
> supposed to know it's about
> > history?
>
> ?? Didn't you yourself say several messages ago it was NOT about
> history?  I quote:
>
> > > Oh, no, this is about sorting completion candidates, not history.
>
> So now you say this IS about history?
>

It's about sorting completion candidates so that their order matches the
order of the minibuffer history.  So yes, it relates to history.  But it's
not sorting the history - that's what I meant.


> But I'm okay with "Sorted in chronological order of minibuffer history".
>
> This is tautology: history is always in chronological order.  We are
> in effect wasting one word by repeating another.
>

Okay, I agree. I think "history" is the more important of the two words
here, so how about "Sorted in order of minibuffer history"

>

[-- Attachment #2: Type: text/html, Size: 3346 bytes --]

Re: Updating *Completions* as you type

[Eli Zaretskii]

> From: Spencer Baugh <sbaugh@janestreet.com>
> Date: Thu, 30 Nov 2023 07:33:01 -0500
> Cc: Juri Linkov <juri@linkov.net>, Spencer Baugh <sbaugh@catern.com>, emacs-devel@gnu.org
> 
>  > But I'm okay with "Sorted in chronological order of minibuffer history".
> 
>  This is tautology: history is always in chronological order.  We are
>  in effect wasting one word by repeating another.
> 
> Okay, I agree. I think "history" is the more important of the two words here, so how about "Sorted in
> order of minibuffer history"

I can live with that.

Re: Updating *Completions* as you type

[Spencer Baugh]

[-- Attachment #1: Type: text/plain, Size: 648 bytes --]


Eli Zaretskii <eliz@gnu.org> writes:
>> From: Spencer Baugh <sbaugh@catern.com>
>> Date: Tue, 28 Nov 2023 14:38:55 +0000 (UTC)
>> Cc: Spencer Baugh <sbaugh@janestreet.com>, emacs-devel@gnu.org
>>    :type '(choice (const :tag "No sorting" nil)
>>                   (const :tag "Alphabetical sorting" alphabetical)
>> +                 (const :tag "Historical sorting" historical)
>                                  ^^^^^^^^^^^^^^^^^^
> "Chronological sorting"

I think "historical sorting" is more intuitive here.  "Chronological"
seems more ambiguous - it could mean something different for files, for
example.

Fixed patch for everything else:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Add-historical-option-to-completions-sort.patch --]
[-- Type: text/x-patch, Size: 6072 bytes --]

From 326da292ef0512a518ec5ed7140c028e53cb6cd7 Mon Sep 17 00:00:00 2001
From: Spencer Baugh <sbaugh@catern.com>
Date: Tue, 17 Oct 2023 09:09:55 -0400
Subject: [PATCH] Add historical option to completions-sort

Support sorting candidates in *Completions* by the order they show up
in the minibuffer history.

Also add minibuffer-sort-alphabetically and
minibuffer-sort-by-history, which are usable for both completions-sort
and display-sort-function.

* lisp/minibuffer.el (completions-sort): Document 'historical option.
(minibuffer-completion-help): Support 'historical option.
(minibuffer-sort-alphabetically)
(minibuffer-completion-base, minibuffer-sort-by-history): Add.
* etc/NEWS: Announce it.
---
 etc/NEWS           |  5 ++++
 lisp/minibuffer.el | 65 +++++++++++++++++++++++++++++++++++++++++-----
 2 files changed, 64 insertions(+), 6 deletions(-)

diff --git a/etc/NEWS b/etc/NEWS
index fd633fad6fb..12a37176ee0 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -620,6 +620,11 @@ completions window.  When the completions window is not visible,
 then all these keys have their usual meaning in the minibuffer.
 This option is supported for in-buffer completion as well.
 
+*** New value 'historical' for user option 'completions-sort'
+When 'completions-sort' is set to 'historical', completion candidates
+will be sorted by their chronological order in the minibuffer history,
+with more recent candidates appearing first.
+
 ** Pcomplete
 
 ---
diff --git a/lisp/minibuffer.el b/lisp/minibuffer.el
index 5c12d9fc914..77703a4e330 100644
--- a/lisp/minibuffer.el
+++ b/lisp/minibuffer.el
@@ -1314,14 +1314,27 @@ completion-cycle-threshold
 (defcustom completions-sort 'alphabetical
   "Sort candidates in the *Completions* buffer.
 
-The value can be nil to disable sorting, `alphabetical' for
-alphabetical sorting or a custom sorting function.  The sorting
-function takes and returns a list of completion candidate
-strings."
+Completion candidates in the *Completions* buffer are sorted
+depending on the value.
+
+If it's nil, sorting is disabled.
+If it's the symbol `alphabetical', candidates are sorted by
+`minibuffer-sort-alphabetically'.
+If it's the symbol `historical', candidates are sorted by
+`minibuffer-sort-by-history'.
+If it's a function, the function is called to sort the candidates.
+The sorting function takes a list of completion candidate
+strings, which it may modify; it should return a sorted list,
+which may be the same.
+
+If the completion-specific metadata provides a
+`display-sort-function', that function overrides the value of
+this variable."
   :type '(choice (const :tag "No sorting" nil)
                  (const :tag "Alphabetical sorting" alphabetical)
+                 (const :tag "Historical sorting" historical)
                  (function :tag "Custom function"))
-  :version "29.1")
+  :version "30.1")
 
 (defcustom completions-group nil
   "Enable grouping of completion candidates in the *Completions* buffer.
@@ -1647,6 +1660,44 @@ minibuffer--sort-preprocess-history
                      (substring c base-size)))
                  hist)))))
 
+(defun minibuffer-sort-alphabetically (completions)
+  "Sort COMPLETIONS alphabetically.
+
+COMPLETIONS are sorted alphabetically by `string-lessp'.
+
+This is a suitable function to use for `completions-sort' or to
+include as `display-sort-function' in completion metadata."
+  (sort completions #'string-lessp))
+
+(defvar minibuffer-completion-base nil
+  "The base for the current completion.
+
+This is the part of the current minibuffer input which comes
+before the current completion field, as determined by
+`completion-boundaries'.  This is primarily relevant for file
+names, where this is the directory component of the file name.")
+
+(defun minibuffer-sort-by-history (completions)
+  "Sort COMPLETIONS by their position in `minibuffer-history-variable'.
+
+COMPLETIONS are sorted first by `minibuffer-sort-alphbetically',
+then any elements occuring in the minibuffer history list are
+moved to the front based on the chronological order they occur in
+the history.  If a history variable hasn't been specified for
+this call of `completing-read', COMPLETIONS are sorted only by
+`minibuffer-sort-alphbetically'.
+
+This is a suitable function to use for `completions-sort' or to
+include as `display-sort-function' in completion metadata."
+  (let ((alphabetized (sort completions #'string-lessp)))
+    ;; Only use history when it's specific to these completions.
+    (if (eq minibuffer-history-variable
+            (default-value minibuffer-history-variable))
+        alphabetized
+      (minibuffer--sort-by-position
+       (minibuffer--sort-preprocess-history minibuffer-completion-base)
+       alphabetized))))
+
 (defun minibuffer--group-by (group-fun sort-fun elems)
   "Group ELEMS by GROUP-FUN and sort groups by SORT-FUN."
   (let ((groups))
@@ -2409,6 +2460,7 @@ minibuffer-completion-help
       (let* ((last (last completions))
              (base-size (or (cdr last) 0))
              (prefix (unless (zerop base-size) (substring string 0 base-size)))
+             (minibuffer-completion-base (substring string 0 base-size))
              (base-prefix (buffer-substring (minibuffer--completion-prompt-end)
                                             (+ start base-size)))
              (base-suffix
@@ -2473,7 +2525,8 @@ minibuffer-completion-help
                                             (funcall sort-fun completions)
                                           (pcase completions-sort
                                             ('nil completions)
-                                            ('alphabetical (sort completions #'string-lessp))
+                                            ('alphabetical (minibuffer-sort-alphabetically completions))
+                                            ('historical (minibuffer-sort-by-history completions))
                                             (_ (funcall completions-sort completions)))))
 
                       ;; After sorting, group the candidates using the
-- 
2.42.1

Re: Updating *Completions* as you type

[Eli Zaretskii]

> From: Spencer Baugh <sbaugh@catern.com>
> Date: Tue, 28 Nov 2023 23:56:14 +0000 (UTC)
> Cc: juri@linkov.net, sbaugh@janestreet.com, emacs-devel@gnu.org
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> >> From: Spencer Baugh <sbaugh@catern.com>
> >> Date: Tue, 28 Nov 2023 14:38:55 +0000 (UTC)
> >> Cc: Spencer Baugh <sbaugh@janestreet.com>, emacs-devel@gnu.org
> >>    :type '(choice (const :tag "No sorting" nil)
> >>                   (const :tag "Alphabetical sorting" alphabetical)
> >> +                 (const :tag "Historical sorting" historical)
> >                                  ^^^^^^^^^^^^^^^^^^
> > "Chronological sorting"
> 
> I think "historical sorting" is more intuitive here.  "Chronological"
> seems more ambiguous - it could mean something different for files, for
> example.

I don't understand why, please explain.  Did history stop changing in
chronological order at some point and I missed that?  That is, doesn't
the first minibuffer input always precede the second one in
chronological order?

Re: Updating *Completions* as you type

[Juri Linkov]

> Fixed patch for everything else:

No more comments in three days, so your latest patch is pushed now, thanks.

Re: Updating *Completions* as you type

[Eli Zaretskii]

> From: Juri Linkov <juri@linkov.net>
> Cc: Eli Zaretskii <eliz@gnu.org>, sbaugh@janestreet.com, emacs-devel@gnu.org
> Date: Sun, 03 Dec 2023 19:25:16 +0200
> 
> > Fixed patch for everything else:
> 
> No more comments in three days, so your latest patch is pushed now, thanks.

The NEWS entry says:

  *** New value 'historical' for user option 'completions-sort'
  When 'completions-sort' is set to 'historical', completion candidates
  will be sorted by their chronological order in the minibuffer history,
  with more recent candidates appearing first.

But if "more recent candidates appear first", this is not the
chronological order, this is the reverse chronological order.  So
which part of the above is correct, the "chronological order" part or
the "more recent candidates first" part?

Btw, the doc string of minibuffer-sort-by-history seems to indicate
that both the above NEWS entry and the doc string of the user options
omit crucial details:

  COMPLETIONS are sorted first by `minibuffer-sort-alphbetically',
  then any elements occuring in the minibuffer history list are
  moved to the front based on the chronological order they occur in
  the history.

This "alphabetical order first" part is not mentioned anywhere else.
It should at least be mentioned in the doc string of the user option
and at least hinted upon in NEWS.

Re: Updating *Completions* as you type

[Juri Linkov]

> The NEWS entry says:
>
>   *** New value 'historical' for user option 'completions-sort'
>   When 'completions-sort' is set to 'historical', completion candidates
>   will be sorted by their chronological order in the minibuffer history,
>   with more recent candidates appearing first.
>
> But if "more recent candidates appear first", this is not the
> chronological order, this is the reverse chronological order.  So
> which part of the above is correct, the "chronological order" part or
> the "more recent candidates first" part?

In fact the word "chronological" bears no significance in this context.
This feature is about the *history* variables, not some "chronological"
variables.  So this should be sufficient:

    *** New value 'historical' for user option 'completions-sort'
    When 'completions-sort' is set to 'historical', completion candidates
    will be sorted by their order in the minibuffer history,
    with more recent candidates appearing first.

> Btw, the doc string of minibuffer-sort-by-history seems to indicate
> that both the above NEWS entry and the doc string of the user options
> omit crucial details:
>
>   COMPLETIONS are sorted first by `minibuffer-sort-alphbetically',
>   then any elements occuring in the minibuffer history list are
>   moved to the front based on the chronological order they occur in
>   the history.
>
> This "alphabetical order first" part is not mentioned anywhere else.
> It should at least be mentioned in the doc string of the user option
> and at least hinted upon in NEWS.

Agreed.

Re: Updating *Completions* as you type

[Juri Linkov]

>>>> This is what I believe they should do: we add a category,
>>>> and they support it as well.
>>>
>>> OK, I'm fine with that, but when we do that, I think the per-table
>>> option should override the per-category option.
>>
>> I agree that the per-table option should override the per-category option,
>> but see no way to distinguish customized values from hard-coded ones
>> without trying to turn all hard-coded values into options.
>
> Right.  I guess my position is that there's not that many hard-coded
> display-sort-functions (only 5 in core Emacs), so turning them all to
> options is fine.

And my position is that simpler to add a category or some identifier.
So we need more opinions.

>> display-buffer has a similar problem, but the difference is that
>> it's possible to identify a buffer by its name and use a regexp
>> to match buffer names.  Whereas for completing-read it's not easy
>> to identify a completion table.  A category matches a set of
>> completion tables, so maybe we need another identification
>> for individual tables?
>
> True, that would help.  Maybe the function symbol for the completion
> table?  Tables are usually lambdas today, but maybe we could make it
> easy to use a defun'd function instead, which would be very good for
> comprehensibility in general IMO.

Turning lambdas into symbols looks good, this is like the existing
'help--symbol-completion-table'.

Turning completion table lambdas into symbols

[Spencer Baugh]

Juri Linkov <juri@linkov.net> writes:
>>> display-buffer has a similar problem, but the difference is that
>>> it's possible to identify a buffer by its name and use a regexp
>>> to match buffer names.  Whereas for completing-read it's not easy
>>> to identify a completion table.  A category matches a set of
>>> completion tables, so maybe we need another identification
>>> for individual tables?
>>
>> True, that would help.  Maybe the function symbol for the completion
>> table?  Tables are usually lambdas today, but maybe we could make it
>> easy to use a defun'd function instead, which would be very good for
>> comprehensibility in general IMO.
>
> Turning lambdas into symbols looks good, this is like the existing
> 'help--symbol-completion-table'.

Like 'help--symbol-completion-table' in what way?

One thing that requires a lambda is when the table is over some custom
data.  But that can sometimes be avoided by moving the logic into the
table, like this for example:

--- a/lisp/progmodes/project.el
+++ b/lisp/progmodes/project.el
@@ -1765,6 +1765,13 @@ project-forget-project
 
 (defvar project--dir-history)
 
+(defun project--project-dir-completion-table (string pred action)
+  (cond
+   ((eq action 'metadata)
+    '(metadata . ((category . project-dir))))
+   (t
+    (complete-with-action action (cons "... (choose a dir)" project--list) string pred))))
+
 (defun project-prompt-project-dir ()
   "Prompt the user for a directory that is one of the known project roots.
 The project is chosen among projects known from the project list,
@@ -1772,18 +1779,14 @@ project-prompt-project-dir
 It's also possible to enter an arbitrary directory not in the list."
   (project--ensure-read-project-list)
   (let* ((dir-choice "... (choose a dir)")
-         (choices
-          ;; XXX: Just using this for the category (for the substring
-          ;; completion style).
-          (project--file-completion-table
-           (append project--list `(,dir-choice))))
          (project--dir-history (project-known-project-roots))
          (pr-dir ""))
     (while (equal pr-dir "")
       ;; If the user simply pressed RET, do this again until they don't.
       (setq pr-dir
             (let (history-add-new-input)
-              (completing-read "Select project: " choices nil t nil 'project--dir-history))))
+              (completing-read "Select project: " #'project--project-dir-completion-table
+                               nil t nil 'project--dir-history))))
     (if (equal pr-dir dir-choice)
         (read-directory-name "Select directory: " default-directory nil t)
       pr-dir)))

The trickier case is when the table actually has some internal state or
calculation which it needs to preserve across multiple calls.  Like I
recently asked about in my recent mail to emacs-devel with subject
"State and caching in completion tables".

If there was a canonical way for a completion table to maintain some
state which *doesn't* require the completion table to be a lambda, I
think most completion tables could become defuns instead of lambdas.
(And then they could be customized based on the function symbol)

Re: Turning completion table lambdas into symbols

[Dmitry Gutov]

On 29/11/2023 01:36, Spencer Baugh wrote:
> If there was a canonical way for a completion table to maintain some
> state which*doesn't*  require the completion table to be a lambda, I
> think most completion tables could become defuns instead of lambdas.
> (And then they could be customized based on the function symbol)

Maybe take a look at how company-capf caches the results of calling a 
CAPF function?

Depending on the data, the cache key is either 
buffer/point/chars-modified-tick tuple, or the "completion session" (but 
in a company-mode specific way).

For CAPF and Company, we also discussed the idea of a "session object" 
some time ago, but that's not in the current API so far.

Re: Turning completion table lambdas into symbols

[Spencer Baugh]

Dmitry Gutov <dmitry@gutov.dev> writes:
> On 29/11/2023 01:36, Spencer Baugh wrote:
>> If there was a canonical way for a completion table to maintain some
>> state which*doesn't*  require the completion table to be a lambda, I
>> think most completion tables could become defuns instead of lambdas.
>> (And then they could be customized based on the function symbol)
>
> Maybe take a look at how company-capf caches the results of calling a
> CAPF function?
>
> Depending on the data, the cache key is either
> buffer/point/chars-modified-tick tuple, or the "completion session"
> (but in a company-mode specific way).

Hm, I think I more want to support state/caching inside the completion
table itself.  Not just caching the output of the completions.  That
allows a lot more flexibility.

For example, project-prompt-project-name could be changed to use a new
completion table defun named project--project-name-completion-table.
project--project-name-completion-table would internally maintain a map
between project directories and project names, which it adds to whenever
completion is requested (instead of creating that map up front in
project-prompt-project-name, as we currently do).  If there was no
support for state between completion table invocations,
project--project-name-completion-table would have to call
project--find-in-directory and project-name for each project on each
completion table invocation, which would be slow.

> For CAPF and Company, we also discussed the idea of a "session object"
> some time ago, but that's not in the current API so far.

Right, I think I'd much rather some kind of "session object" at the
level of completing-read/the programmed completion API.

Do you have a link to the previous discussion?

I haven't thought too much about it, but maybe some new dynamic variable
which is bound to nil at the top level of completing-read-default, so
the completion table can change it over time and preserve state through
the course of the completion.  Works fine with nested completing-reads.

Alternatively, if today a completion table is always invoked with the
same current buffer (probably the case?), we could formalize that and
just let a completion table store state in buffer-local variables.
Maybe with some new 'initialize operation in completion tables which is
called when a new completing-read starts, or maybe the completion table
can just detect that somehow and initialize the variables itself.  That
is nicer than having a single variable which all completion tables
share.  However, completion table writers would need to be careful not
to break when doing nested completing-reads on the same table.

Re: Turning completion table lambdas into symbols

[Dmitry Gutov]

On 29/11/2023 21:26, Spencer Baugh wrote:
>> For CAPF and Company, we also discussed the idea of a "session object"
>> some time ago, but that's not in the current API so far.
> Right, I think I'd much rather some kind of "session object" at the
> level of completing-read/the programmed completion API.
> 
> Do you have a link to the previous discussion?

It's been here and there, but the last mention was right in the middle 
of this:

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=47711#272
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=47711#320

> I haven't thought too much about it, but maybe some new dynamic variable
> which is bound to nil at the top level of completing-read-default, so
> the completion table can change it over time and preserve state through
> the course of the completion.  Works fine with nested completing-reads.

That could work. If you further progress on this (in a new thread?), I 
suggest you tag all the interested parties (or include Stefan, at least).

> Alternatively, if today a completion table is always invoked with the
> same current buffer (probably the case?), we could formalize that and
> just let a completion table store state in buffer-local variables.
> Maybe with some new 'initialize operation in completion tables which is
> called when a new completing-read starts, or maybe the completion table
> can just detect that somehow and initialize the variables itself.  That
> is nicer than having a single variable which all completion tables
> share.  However, completion table writers would need to be careful not
> to break when doing nested completing-reads on the same table.

It's not rocket science, but the hard part is to pick a protocol that's 
backward-compatible and weave it through all the related pieces of code.

Also see the note about backspacing around here: 
https://github.com/minad/corfu/wiki#configuring-corfu-for-eglot (where 
doing an edit before the current BEG doesn't abort completion but does 
make the cache invalid), but that might be something to handle in the 
caching logic anyway.

Re: Turning completion table lambdas into symbols

[Juri Linkov]

>>>> display-buffer has a similar problem, but the difference is that
>>>> it's possible to identify a buffer by its name and use a regexp
>>>> to match buffer names.  Whereas for completing-read it's not easy
>>>> to identify a completion table.  A category matches a set of
>>>> completion tables, so maybe we need another identification
>>>> for individual tables?
>>>
>>> True, that would help.  Maybe the function symbol for the completion
>>> table?  Tables are usually lambdas today, but maybe we could make it
>>> easy to use a defun'd function instead, which would be very good for
>>> comprehensibility in general IMO.
>>
>> Turning lambdas into symbols looks good, this is like the existing
>> 'help--symbol-completion-table'.
>
> Like 'help--symbol-completion-table' in what way?

Just an example where minibuffer-completion-table is a symbol.

> One thing that requires a lambda is when the table is over some custom
> data.  But that can sometimes be avoided by moving the logic into the
> table, like this for example:
>
> +(defun project--project-dir-completion-table (string pred action)
> +  (cond
> +   ((eq action 'metadata)
> +    '(metadata . ((category . project-dir))))
> +   (t
> +    (complete-with-action action (cons "... (choose a dir)" project--list) string pred))))
> ...
> -              (completing-read "Select project: " choices nil t nil 'project--dir-history))))
> +              (completing-read "Select project: " #'project--project-dir-completion-table
> +                               nil t nil 'project--dir-history))))

> If there was a canonical way for a completion table to maintain some
> state which *doesn't* require the completion table to be a lambda, I
> think most completion tables could become defuns instead of lambdas.
> (And then they could be customized based on the function symbol)

Or they could be customized based on a new symbol for table identification
without turning lambdas into defuns:

  (completing-read "Select project: "
                   (lambda (string pred action)
                     (cond
                      ((eq action 'metadata)
                       '(metadata
                         . ((category . project-dir)
                            (table . project-dir-completion-table))))
                      (t
                       (complete-with-action action choices string pred))))
                   nil t nil 'project--dir-history)

Re: Updating *Completions* as you type

[John Yates]

[-- Attachment #1: Type: text/plain, Size: 1273 bytes --]

Sharing this insight in the associated documentation will aid users.

On Tue, Nov 21, 2023 at 3:31 AM Juri Linkov <juri@linkov.net> wrote:

> > I expect some disagreement about the following question: should
> > completion-category-overrides override the display-sort-function
> > returned by the completion table?  That is, should it instead be:
> >
> > (or
> >  (alist-get 'display-sort-function (alist-get category
> completion-category-overrides))
> >  (alist-get 'display-sort-function metadata)
> >  (alist-get 'display-sort-function (alist-get category
> completion-category-defaults))
> >  completions-sort)
>
> I think this is the most correct precedence since the users should be
> able to override the function call metadata.  We have an analogous
> priority levels for 'display-buffer':
>
> 1. display-buffer-overriding-action
> 2. display-buffer-alist
> 3. function call arguments that correspond to completion metadata
> 4. display-buffer-base-action
> 5. display-buffer-fallback-action
>
> Since completion-category-overrides is a user option
> it corresponds to display-buffer-alist.
> And completion-category-defaults looks like
> display-buffer-base-action.
>
>

-- 
John Yates
505 Tremont St, #803
Boston, MA 02116

[-- Attachment #2: Type: text/html, Size: 1926 bytes --]

Re: Updating *Completions* as you type

[Juri Linkov]

>> 1. display-buffer-overriding-action
>> 2. display-buffer-alist
>> 3. function call arguments that correspond to completion metadata
>> 4. display-buffer-base-action
>> 5. display-buffer-fallback-action

> Sharing this insight in the associated documentation will aid users.

This is thoroughly documented in (info "(elisp) Choosing Window"):

     ‘display-buffer’ builds a list of action functions and an action
     alist, by consolidating display actions from the following sources
     (in order of their precedence, from highest to lowest):

        • The variable ‘display-buffer-overriding-action’.
        • The user option ‘display-buffer-alist’.
        • The ACTION argument.
        • The user option ‘display-buffer-base-action’.
        • The constant ‘display-buffer-fallback-action’.

Re: Updating *Completions* as you type

[John Yates]

On Tue, Nov 21, 2023 at 12:12 PM Juri Linkov <juri@linkov.net> wrote:
>
> > Sharing this insight in the associated documentation will aid users.
>
> This is thoroughly documented in (info "(elisp) Choosing Window"):

I was insufficiently clear.  I meant the emerging parallel ordering of
completion-category controls and display-buffer control:

> Since completion-category-overrides is a user option
> it corresponds to display-buffer-alist.
> And completion-category-defaults looks like
> display-buffer-base-action.

Re: Updating *Completions* as you type

[Juri Linkov]

> Also: it might be nice to switch between history-sorting and
> alphabetized-sorting during the course of completion, both for files and
> buffers.  Maybe we could do that by making a command which puts
> completion into a mode where it just ignores the display-sort-function
> specified by the completion metadata, and just always uses
> completions-sort.

To be able to implement switching between different sort orders
with a new key, first we need to implement revert-buffer-function
(bound to 'g') in the *Completions* buffer.

Then it will be possible just to toggle a new buffer-local variable
(initially set from 'completions-sort') and to revert the *Completions*
buffer with a new sort order.  Also will need to keep a previously
selected candidate that is implemented by your recent patch.

Re: Updating *Completions* as you type

[sbaugh]

Tangentially, Juri, I have a thought about a redesign of
minibuffer-next-completion and minibuffer-choose-completion.

What if the concept of "current selected completion" was unified with
"the default completion"?  This could be a nice, general UI.

Specifically, with switch-to-buffer and a default of init.el:

- If init.el is present in *Completions*, start out with point on it.
  This would be purely a display nicety, it wouldn't actually affect
  anything yet.  (This would be easy with my patch which I posted
  elsewhere in this thread to preserve the location of point in
  *Completions*)
- If and when the user invokes minibuffer-next-completion:
  - The default changes to whatever the new selected completion is
  - The prompt text "(default init.el)" changes permanently to literally
    "(default selected completion)"
- RET, as always, chooses the default if the minibuffer is empty; if the
  user has done minibuffer-next-completion, the default is the selected
  completion, so RET will choose that.
- M-RET (minibuffer-choose-completion) is replaced with a new command
  which immediately chooses the default, whatever it is, ignoring the
  current contents of the minibuffer
- C-u M-RET inserts the default in the the minibuffer, without exiting
  (matching the behavior of C-u minibuffer-choose-completion)

I think this has some nice benefits in reducing the number of concepts
people need to track.  If the minibuffer is empty, they can just use
minibuffer-next-completion a few times followed by RET to select a
completion, no need to use M-RET.  Plus, the new M-RET and C-u M-RET
would be useful even to users who don't use minibuffer-next-completion.

I also think this would make it less painful to set
minibuffer-completion-auto-choose to nil, which matches
completion-in-region better and also works much better with
completions-auto-update.

Re: Updating *Completions* as you type

[Juri Linkov]

> What if the concept of "current selected completion" was unified with
> "the default completion"?  This could be a nice, general UI.
>
> Specifically, with switch-to-buffer and a default of init.el:
>
> - If init.el is present in *Completions*, start out with point on it.
>   This would be purely a display nicety, it wouldn't actually affect
>   anything yet.  (This would be easy with my patch which I posted
>   elsewhere in this thread to preserve the location of point in
>   *Completions*)

I think preselecting the default value in the middle of completions
would make sense only when completions were sorted by the order of the
list of default values (from M-n M-n ...).  Then the first default value
would be at the top of completions, and it would easier for users
to navigate completions top-down.

> - If and when the user invokes minibuffer-next-completion:
>   - The default changes to whatever the new selected completion is
>   - The prompt text "(default init.el)" changes permanently to literally
>     "(default selected completion)"

Changing the prompt might interfere with such packages as minibuf-eldef.el
and other cases of customized minibuffer-default-prompt-format.
Also this might break commands that manually handle the default value
for empty input.

> - RET, as always, chooses the default if the minibuffer is empty; if the
>   user has done minibuffer-next-completion, the default is the selected
>   completion, so RET will choose that.
> - M-RET (minibuffer-choose-completion) is replaced with a new command
>   which immediately chooses the default, whatever it is, ignoring the
>   current contents of the minibuffer

What should RET do after the user navigated in *Completions* and
switched back to the minibuffer.  A different candidate is highlighted
in *Completions*, while the default value remains unchanged.

BTW, while looking at this case I found a problem with your first patch:
after navigating in *Completions* and switching back to the minibuffer
point is reset to the beginning of the *Completions* buffer.

> - C-u M-RET inserts the default in the the minibuffer, without exiting
>   (matching the behavior of C-u minibuffer-choose-completion)

Usually the default is inserted by M-n.

> I think this has some nice benefits in reducing the number of concepts
> people need to track.  If the minibuffer is empty, they can just use
> minibuffer-next-completion a few times followed by RET to select a
> completion, no need to use M-RET.  Plus, the new M-RET and C-u M-RET
> would be useful even to users who don't use minibuffer-next-completion.

It seems this is intended to solve the problem of a mismatch between
the highlighted candidate and the contents of the minibuffer?
Such problem exists, for example, in icomplete-mode where
RET returns the contents of the minibuffer, so a special key 'C-j'
is dedicated to select the highlighted candidate.  For selecting
a highlighted candidate from *Completions* such key is 'M-RET'.

> I also think this would make it less painful to set
> minibuffer-completion-auto-choose to nil, which matches
> completion-in-region better and also works much better with
> completions-auto-update.

Sorry, I don't understand how this would make
minibuffer-completion-auto-choose=nil less painful.

Since there is no concept of the default value for completion-in-region,
'M-RET' is the only way to choose the highlighted candidate.

Re: Updating *Completions* as you type

[Spencer Baugh]

Juri Linkov <juri@linkov.net> writes:
>> What if the concept of "current selected completion" was unified with
>> "the default completion"?  This could be a nice, general UI.
>>
>> Specifically, with switch-to-buffer and a default of init.el:
>>
>> - If init.el is present in *Completions*, start out with point on it.
>>   This would be purely a display nicety, it wouldn't actually affect
>>   anything yet.  (This would be easy with my patch which I posted
>>   elsewhere in this thread to preserve the location of point in
>>   *Completions*)
>
> I think preselecting the default value in the middle of completions
> would make sense only when completions were sorted by the order of the
> list of default values (from M-n M-n ...).  Then the first default value
> would be at the top of completions, and it would easier for users
> to navigate completions top-down.

Yes, that was the case that inspired this specific idea: the case where
completions are sorted by the list of defaults, and so the default value
is the first completion.

But it occured to me that there's no reason to limit to just the case
where the default value is the first completion.  We can just always
move point to the default wherever it occurs in the completions, as long
as it occurs somewhere.  Seems like a nice improvement.

Although I guess it does cause M-<down> to no longer go to the first
completion, and M-<up> to no longer go to the last one.  Which is a bit
annoying.

Maybe we should have bindings to move to the first and last completions.
Whatever they are, they should also work in completion-in-region-mode,
though, so M-< and M-> definitely won't work... Maybe M-0 M-<down> to go
to the first, and M-0 M-<up> to go to the last?  (Or vice versa?)

>> - If and when the user invokes minibuffer-next-completion:
>>   - The default changes to whatever the new selected completion is
>>   - The prompt text "(default init.el)" changes permanently to literally
>>     "(default selected completion)"
>
> Changing the prompt might interfere with such packages as minibuf-eldef.el
> and other cases of customized minibuffer-default-prompt-format.

I think as long as we use format-prompt, minibuf-eldef.el and customized
values of minibuffer-default-prompt-format will still work.  Something
to be careful about though.  Changing the prompt text isn't essential,
anyway - it may just be a bit confusing if it's not changed.

> Also this might break commands that manually handle the default value
> for empty input.

As long as the default value is present in the completions, the user can
always select it again.  Or the user can always just do M-n to select
the original default value - I don't expect this would change that.

>> - RET, as always, chooses the default if the minibuffer is empty; if the
>>   user has done minibuffer-next-completion, the default is the selected
>>   completion, so RET will choose that.
>> - M-RET (minibuffer-choose-completion) is replaced with a new command
>>   which immediately chooses the default, whatever it is, ignoring the
>>   current contents of the minibuffer
>
> What should RET do after the user navigated in *Completions* and
> switched back to the minibuffer.  A different candidate is highlighted
> in *Completions*, while the default value remains unchanged.

RET should probably choose the highlighted candidate in *Completions* in
that case.

BTW to be clear I don't mean that we would actually change
minibuffer-default; just that RET with an empty minibuffer would provide
the current selected completion rather than minibuffer-default.

> BTW, while looking at this case I found a problem with your first patch:
> after navigating in *Completions* and switching back to the minibuffer
> point is reset to the beginning of the *Completions* buffer.

Good catch!  If you see my other patch which I posted in this thread,
"Keep point on the same completion in the completions buffer", that is a
nice orthogonal improvement which incidentally also fixes that bug in
completions-auto-update.

>> - C-u M-RET inserts the default in the the minibuffer, without exiting
>>   (matching the behavior of C-u minibuffer-choose-completion)
>
> Usually the default is inserted by M-n.

True, and I don't expect to change that.  I guess that's probably
sufficient.

OK, so maybe the thing I want to propose is not "we'll change the
default to whatever the current selected completion is" but instead "RET
with an empty minibuffer will submit the current selected completion".
That's equivalent, if I drop the ideas of changing the prompt and
changing M-RET.

>> I think this has some nice benefits in reducing the number of concepts
>> people need to track.  If the minibuffer is empty, they can just use
>> minibuffer-next-completion a few times followed by RET to select a
>> completion, no need to use M-RET.  Plus, the new M-RET and C-u M-RET
>> would be useful even to users who don't use minibuffer-next-completion.
>
> It seems this is intended to solve the problem of a mismatch between
> the highlighted candidate and the contents of the minibuffer?
> Such problem exists, for example, in icomplete-mode where
> RET returns the contents of the minibuffer, so a special key 'C-j'
> is dedicated to select the highlighted candidate.  For selecting
> a highlighted candidate from *Completions* such key is 'M-RET'.

True, good analysis.

Specifically though it's about the case when the minibuffer is empty.  I
think it would be nice for RET to submit the highlighted candidate in
that case, if there is one.

That matches icomplete-mode's behavior, actually, which is nice.

>> I also think this would make it less painful to set
>> minibuffer-completion-auto-choose to nil, which matches
>> completion-in-region better and also works much better with
>> completions-auto-update.
>
> Sorry, I don't understand how this would make
> minibuffer-completion-auto-choose=nil less painful.
>
> Since there is no concept of the default value for completion-in-region,
> 'M-RET' is the only way to choose the highlighted candidate.

Eh, this aspect is probably specific to me.

Re: Updating *Completions* as you type

[Spencer Baugh]

Spencer Baugh <sbaugh@catern.com> writes:
> That matches icomplete-mode's behavior, actually, which is nice.

Oh, actually it doesn't.  It matches ido-mode and fido-mode and a host
of completion mechanisms outside core, though.  I still think it's
desirable, at least as a user option.

Re: Updating *Completions* as you type

[Juri Linkov]

>> I think preselecting the default value in the middle of completions
>> would make sense only when completions were sorted by the order of the
>> list of default values (from M-n M-n ...).  Then the first default value
>> would be at the top of completions, and it would easier for users
>> to navigate completions top-down.
>
> Yes, that was the case that inspired this specific idea: the case where
> completions are sorted by the list of defaults, and so the default value
> is the first completion.
>
> But it occured to me that there's no reason to limit to just the case
> where the default value is the first completion.  We can just always
> move point to the default wherever it occurs in the completions, as long
> as it occurs somewhere.  Seems like a nice improvement.
>
> Although I guess it does cause M-<down> to no longer go to the first
> completion, and M-<up> to no longer go to the last one.  Which is a bit
> annoying.

Indeed.

> Maybe we should have bindings to move to the first and last completions.
> Whatever they are, they should also work in completion-in-region-mode,
> though, so M-< and M-> definitely won't work... Maybe M-0 M-<down> to go
> to the first, and M-0 M-<up> to go to the last?  (Or vice versa?)

M-< and M-> work in the minibuffer as well as in the normal buffer.
But they don't work with your first patch ;-)

>> BTW, while looking at this case I found a problem with your first patch:
>> after navigating in *Completions* and switching back to the minibuffer
>> point is reset to the beginning of the *Completions* buffer.
>
> Good catch!  If you see my other patch which I posted in this thread,
> "Keep point on the same completion in the completions buffer", that is a
> nice orthogonal improvement which incidentally also fixes that bug in
> completions-auto-update.

Your third patch adds a new feature that just hides the bug by forcing
the previous position over the wrong new one.  I think the proper fix
would be to add more commands to completions-no-auto-update-commands
in your first patch.  Or to change it from opt-out to opt-in by renaming
to completions-auto-update-commands.

> OK, so maybe the thing I want to propose is not "we'll change the
> default to whatever the current selected completion is" but instead "RET
> with an empty minibuffer will submit the current selected completion".
> That's equivalent, if I drop the ideas of changing the prompt and
> changing M-RET.

Agreed, I think the idea of changing the prompt on the fly won't fly.

>>> I think this has some nice benefits in reducing the number of concepts
>>> people need to track.  If the minibuffer is empty, they can just use
>>> minibuffer-next-completion a few times followed by RET to select a
>>> completion, no need to use M-RET.  Plus, the new M-RET and C-u M-RET
>>> would be useful even to users who don't use minibuffer-next-completion.
>>> True, good analysis.
>>
>> Specifically though it's about the case when the minibuffer is empty.  I
>> think it would be nice for RET to submit the highlighted candidate in
>> that case, if there is one.
>>
>> That matches icomplete-mode's behavior, actually, which is nice.
>
> Oh, actually it doesn't.  It matches ido-mode and fido-mode and a host
> of completion mechanisms outside core, though.  I still think it's
> desirable, at least as a user option.

But then such idiosyncrasy of fido-mode causes a lot of bug reports
like bug#55800.

Re: Updating *Completions* as you type

[Spencer Baugh]

Juri Linkov <juri@linkov.net> writes:

>>> I think preselecting the default value in the middle of completions
>>> would make sense only when completions were sorted by the order of the
>>> list of default values (from M-n M-n ...).  Then the first default value
>>> would be at the top of completions, and it would easier for users
>>> to navigate completions top-down.
>>
>> Yes, that was the case that inspired this specific idea: the case where
>> completions are sorted by the list of defaults, and so the default value
>> is the first completion.
>>
>> But it occured to me that there's no reason to limit to just the case
>> where the default value is the first completion.  We can just always
>> move point to the default wherever it occurs in the completions, as long
>> as it occurs somewhere.  Seems like a nice improvement.
>>
>> Although I guess it does cause M-<down> to no longer go to the first
>> completion, and M-<up> to no longer go to the last one.  Which is a bit
>> annoying.
>
> Indeed.
>
>> Maybe we should have bindings to move to the first and last completions.
>> Whatever they are, they should also work in completion-in-region-mode,
>> though, so M-< and M-> definitely won't work... Maybe M-0 M-<down> to go
>> to the first, and M-0 M-<up> to go to the last?  (Or vice versa?)
>
> M-< and M-> work in the minibuffer as well as in the normal buffer.
> But they don't work with your first patch ;-)

Right right, I was just saying one reason we can't use M-< and M-> for
said bindings.

Anyway, I realized that we already have a nice way to move to the first
and last completions: ? M-<up> and ? M-<down>.

That is, run minibuffer-completion-help to recreate *Completions* and
wipe out the current value of point, then go down or up to access the
first or last.

Seems great so let's just stick with that.

>>> BTW, while looking at this case I found a problem with your first patch:
>>> after navigating in *Completions* and switching back to the minibuffer
>>> point is reset to the beginning of the *Completions* buffer.
>>
>> Good catch!  If you see my other patch which I posted in this thread,
>> "Keep point on the same completion in the completions buffer", that is a
>> nice orthogonal improvement which incidentally also fixes that bug in
>> completions-auto-update.
>
> Your third patch adds a new feature that just hides the bug by forcing
> the previous position over the wrong new one.  I think the proper fix
> would be to add more commands to completions-no-auto-update-commands
> in your first patch.  Or to change it from opt-out to opt-in by renaming
> to completions-auto-update-commands.

I would much rather not list commands as opt-in or opt-out at all, I'd
rather all commands just work without special cases.  So I'm working on
features to allow that.

I don't agree with the characterization of "it just hides the bug".  The
bug is that point in *Completions* gets wiped out by auto-updating.  My
third patch preserves point across auto-updating.  That's very directly
solving the bug.

>> OK, so maybe the thing I want to propose is not "we'll change the
>> default to whatever the current selected completion is" but instead "RET
>> with an empty minibuffer will submit the current selected completion".
>> That's equivalent, if I drop the ideas of changing the prompt and
>> changing M-RET.
>
> Agreed, I think the idea of changing the prompt on the fly won't fly.
>
>>>> I think this has some nice benefits in reducing the number of concepts
>>>> people need to track.  If the minibuffer is empty, they can just use
>>>> minibuffer-next-completion a few times followed by RET to select a
>>>> completion, no need to use M-RET.  Plus, the new M-RET and C-u M-RET
>>>> would be useful even to users who don't use minibuffer-next-completion.
>>>> True, good analysis.
>>>
>>> Specifically though it's about the case when the minibuffer is empty.  I
>>> think it would be nice for RET to submit the highlighted candidate in
>>> that case, if there is one.
>>>
>>> That matches icomplete-mode's behavior, actually, which is nice.
>>
>> Oh, actually it doesn't.  It matches ido-mode and fido-mode and a host
>> of completion mechanisms outside core, though.  I still think it's
>> desirable, at least as a user option.
>
> But then such idiosyncrasy of fido-mode causes a lot of bug reports
> like bug#55800.

But we would not have those kinds of issues because when completion
starts, by default, there is no highlighted candidate.

Re: Updating *Completions* as you type

[Juri Linkov]

>>>> BTW, while looking at this case I found a problem with your first patch:
>>>> after navigating in *Completions* and switching back to the minibuffer
>>>> point is reset to the beginning of the *Completions* buffer.
>>>
>>> Good catch!  If you see my other patch which I posted in this thread,
>>> "Keep point on the same completion in the completions buffer", that is a
>>> nice orthogonal improvement which incidentally also fixes that bug in
>>> completions-auto-update.
>>
>> Your third patch adds a new feature that just hides the bug by forcing
>> the previous position over the wrong new one.  I think the proper fix
>> would be to add more commands to completions-no-auto-update-commands
>> in your first patch.  Or to change it from opt-out to opt-in by renaming
>> to completions-auto-update-commands.
>
> I would much rather not list commands as opt-in or opt-out at all, I'd
> rather all commands just work without special cases.  So I'm working on
> features to allow that.
>
> I don't agree with the characterization of "it just hides the bug".  The
> bug is that point in *Completions* gets wiped out by auto-updating.  My
> third patch preserves point across auto-updating.  That's very directly
> solving the bug.

I agree it's helpful to keep point on the selected candidate while
adding characters to the completion.  But the problem is that point
is kept for too long that affects later unralated commands.

>>>>> I think this has some nice benefits in reducing the number of concepts
>>>>> people need to track.  If the minibuffer is empty, they can just use
>>>>> minibuffer-next-completion a few times followed by RET to select a
>>>>> completion, no need to use M-RET.  Plus, the new M-RET and C-u M-RET
>>>>> would be useful even to users who don't use minibuffer-next-completion.
>>>>> True, good analysis.
>>>>
>>>> Specifically though it's about the case when the minibuffer is empty.  I
>>>> think it would be nice for RET to submit the highlighted candidate in
>>>> that case, if there is one.
>>>>
>>>> That matches icomplete-mode's behavior, actually, which is nice.
>>>
>>> Oh, actually it doesn't.  It matches ido-mode and fido-mode and a host
>>> of completion mechanisms outside core, though.  I still think it's
>>> desirable, at least as a user option.
>>
>> But then such idiosyncrasy of fido-mode causes a lot of bug reports
>> like bug#55800.
>
> But we would not have those kinds of issues because when completion
> starts, by default, there is no highlighted candidate.

And the issue occurs as soon as the first candidate is highlighted.

Re: Updating *Completions* as you type

[Spencer Baugh]

Juri Linkov <juri@linkov.net> writes:
>>>>>> I think this has some nice benefits in reducing the number of concepts
>>>>>> people need to track.  If the minibuffer is empty, they can just use
>>>>>> minibuffer-next-completion a few times followed by RET to select a
>>>>>> completion, no need to use M-RET.  Plus, the new M-RET and C-u M-RET
>>>>>> would be useful even to users who don't use minibuffer-next-completion.
>>>>>> True, good analysis.
>>>>>
>>>>> Specifically though it's about the case when the minibuffer is empty.  I
>>>>> think it would be nice for RET to submit the highlighted candidate in
>>>>> that case, if there is one.
>>>>>
>>>>> That matches icomplete-mode's behavior, actually, which is nice.
>>>>
>>>> Oh, actually it doesn't.  It matches ido-mode and fido-mode and a host
>>>> of completion mechanisms outside core, though.  I still think it's
>>>> desirable, at least as a user option.
>>>
>>> But then such idiosyncrasy of fido-mode causes a lot of bug reports
>>> like bug#55800.
>>
>> But we would not have those kinds of issues because when completion
>> starts, by default, there is no highlighted candidate.
>
> And the issue occurs as soon as the first candidate is highlighted.

Yes, but that's something the user explicitly chooses to do.  (as long
as we don't preselect completions, which (notwithstanding my patch which
does that) I don't think we should do)

And even if they highlight a candidate, they can always unhighlight by
running minibuffer-completion-help (? or M-?) again.

That's a big advantage we have over fido-mode and icomplete, which lets
us avoid this class of problems: We have a way to represent the state
where no candidate is selected/highlighted.

RE: [External] : Re: Updating *Completions* as you type

[Drew Adams]

> Maybe a new function should allow any sorting order?

FWIW:

Icicles lets a caller of a function such as
`completing-read' specify a set of sort functions
and the initial one to use.

And users can change among those sort functions
on the fly - by cycling them or inputting the name
(with completion of course).  Reverse the current
sort order with a numeric prefix arg.

User switching of sort order is _very_ useful, IMO.
(And yes, incremental updating of *Completions"
needs to automatically re-sort.)
___

https://www.emacswiki.org/emacs/Icicles_-_Sorting_Candidates

______________________________________________

FWIW2:

In library `sortie.el' I implemented the same thing, 
without Icicles, using only vanilla Emacs completion
metadata entries `display-sort-function' and
`cycle-sort-function'.

[I don't know of another library (besides Icicles)
that provides interactive changing of sort order -
but maybe there is one now.]
___

https://www.emacswiki.org/emacs/Sortie

Re: zcomplete

[Philip Kaludercic]

Juri Linkov <juri@linkov.net> writes:

> diff --git a/lisp/zcomplete.el b/lisp/zcomplete.el
> new file mode 100644
> index 00000000000..75a40c0afd3
> --- /dev/null
> +++ b/lisp/zcomplete.el
> @@ -0,0 +1,317 @@
> +;;; zcomplete.el --- zsh-like minibuffer completion based on icomplete -*- lexical-binding: t -*-

I have been trying out zcomplete for the last week, and it works really
well.  If there is no plan to add it to the core, would you at least be
interested in having it added to GNU ELPA?

Re: zcomplete

[Juri Linkov]

>> diff --git a/lisp/zcomplete.el b/lisp/zcomplete.el
>> new file mode 100644
>> index 00000000000..75a40c0afd3
>> --- /dev/null
>> +++ b/lisp/zcomplete.el
>> @@ -0,0 +1,317 @@
>> +;;; zcomplete.el --- zsh-like minibuffer completion based on icomplete -*- lexical-binding: t -*-
>
> I have been trying out zcomplete for the last week, and it works really
> well.  If there is no plan to add it to the core, would you at least be
> interested in having it added to GNU ELPA?

Actually, it has many problems.  Hopefully, we will fix them in the core.
I'd rather not proliferate the number of similar packages by adding
yet another package that does the same as existing mct, vcomplete.

Re: zcomplete

[Protesilaos Stavrou]

> From: Juri Linkov <juri@linkov.net>
> Date: Sun, 22 Oct 2023 20:28:01 +0300

> [... 9 lines elided]

>> I have been trying out zcomplete for the last week, and it works really
>> well.  If there is no plan to add it to the core, would you at least be
>> interested in having it added to GNU ELPA?
>
> Actually, it has many problems.  Hopefully, we will fix them in the core.
> I'd rather not proliferate the number of similar packages by adding
> yet another package that does the same as existing mct, vcomplete.

Just to note that I am interested in what you do here and am happy to
remove mct altogether.

-- 
Protesilaos Stavrou
https://protesilaos.com

Re: zcomplete

[Juri Linkov]

>>> I have been trying out zcomplete for the last week, and it works really
>>> well.  If there is no plan to add it to the core, would you at least be
>>> interested in having it added to GNU ELPA?
>>
>> Actually, it has many problems.  Hopefully, we will fix them in the core.
>> I'd rather not proliferate the number of similar packages by adding
>> yet another package that does the same as existing mct, vcomplete.
>
> Just to note that I am interested in what you do here and am happy to
> remove mct altogether.

No need to remove mct.  The core should provide the basic building blocks.
Then external packages could build more feature-rich experience.
OTOH, in the process of mutual enrichment often external packages act as
a playing ground for new features, and when they mature, some of their
features are included in the core.  Even in this case the package could
continue existence with more peripheral features that satisfy even
more demanding users.

Re: Updating *Completions* as you type

[Daniel Semyonov]

[-- Attachment #1: Type: text/plain, Size: 2403 bytes --]

My package Vcomplete implements this functionality, however I stopped developing it since I reached the conclusion that this isn’t a very useful feature (to me) and all other features of the package were implemented in some form in Emacs.  It still works though, and will use the base Emacs implementation for those other features if they’re available.

Some notes:
- Using after-change-functions to implement this can cause weird issues in packages like Tramp and embark which modify the minibuffer during completion.
- If there are a lot of completions, the completions buffer can take a while to update, and you need to make sure this process blocks user input as little as possible.
- I recall having issues with the completion buffer popping up randomly when typing after completing in a buffer, but I don’t remember why it happened and how I solved it.
- There is also the package aggressive-completion which does something similar.

Daniel

(Sent from a phone, sorry for any weird formatting)

> On 13 Oct 2023, at 8:31, sbaugh@catern.com wrote:
> 
> 
> It would be nice if there was a built-in customization which caused
> *Completions* to update as you type, as long as that buffer is visible.
> I imagine such a request has been made before, so what is the obstacle
> to adding it?
> 
> I would like to figure out a solution which everyone is happy with, and
> then I would be happy to implement it.  It seems to me that it's just a
> matter of, after each keystroke, triggering (for minibuffer completion)
> minibuffer-completion-help or (for buffer completion) some new function
> to populate *Completions* with all-completions output.  This could (I
> guess) be done with after-change-functions, although maybe others have a
> better idea.
> 
> (Btw, if we had this behavior, it also seems like it would help with
> another long-time request: asynchronous completion support.  A
> programmed completion table could internally do something asynchronous
> and stateful, and accumulate results over time, and return more and more
> results each time Emacs calls all-completions/try-completion.  If Emacs
> automatically called all-completions with such a programmed completion
> table, the resulting behavior would be an acceptable approximation of
> asynchronous completion, without having to complicate the programmed
> completion API.)
> 
> 

[-- Attachment #2: Type: text/html, Size: 3419 bytes --]

Re: Updating *Completions* as you type

[Spencer Baugh]

Daniel Semyonov <daniel@dsemy.com> writes:
> My package Vcomplete implements this functionality, however I stopped
> developing it since I reached the conclusion that this isn’t a very
> useful feature (to me) and all other features of the package were
> implemented in some form in Emacs.  It still works though, and will
> use the base Emacs implementation for those other features if they’re
> available.

Oh yes, I did see your package.  One different is that I don't want to
automatically open the *Completions* buffer, just auto-update after it's
open.  Deciding when to open *Completions* is the responsibility of
completion-auto-help and other such options.

> Some notes:
> - Using after-change-functions to implement this can cause weird
> issues in packages like Tramp and embark which modify the minibuffer
> during completion.

Good to know.

> - If there are a lot of completions, the completions buffer can take a
> while to update, and you need to make sure this process blocks user
> input as little as possible.

Hm, perhaps I can optimize the updating of *Completions*, then.  That
would be generally useful.  Maybe we can find a way to allow user input
to continue while minibuffer-completion-help is running.

> - I recall having issues with the completion buffer popping up
> randomly when typing after completing in a buffer, but I don’t
> remember why it happened and how I solved it.

Plausibly this won't be an issue since I don't plan to open
*Completions* if it's not already open.

> - There is also the package aggressive-completion which does something
> similar.

True!  Seems very similar to vcomplete.  Again it has more features than
I want to implement, though.

RE: [External] : Re: Updating *Completions* as you type

[Drew Adams]

FWIW:

I'll mention some similar features in Icicles.  They
were introduced together from the outset (in 2005!),
and have proven useful.

Such features really should be flexible - in no way
hard-coded.  Flexible for use interactively and with 
Lisp.  Don't just look for a good compromise 
behavior - do that for the default behavior, sure.
But let users easily switch behaviors on the fly.
And let Lisp code easily choose different default
(i.e., initial) behaviors for different 
contexts/commands.

> I don't want to automatically open the *Completions*
> buffer, just auto-update after it's open.  Deciding
> when to open *Completions* is the responsibility of
> completion-auto-help and other such options.

Icicles introduced incremental *Completions*-display
updating, providing these options to let users and code
easily control the behavior:

1. `icicle-incremental-completion':

   * `t'  : Update *Completions* incrementally only if 
            it's already displayed.  (Default.)
   * `nil': Don't update *Completions* incrementally.
   * other: Show *Completions* if not shown (+ update).

   `C-#' in the minibuffer cycles the values, so you
   can change the behavior on the fly.

   Some commands, such as those for navigation/search
   choices or those effectively providing multiple-choice
   menus, may want to bind the var to `always', to show
   you what's available at the outset, i.e., the full
   completion domain.

   A command that has many potential candidates (domain)
   might want to bind `icicle-incremental-completion' to
   `nil' at the outset if their display is too costly.

(You can always use `C-#' to
   change.)

2. `icicle-show-Completions-initially-flag':

   Non-nil: Show `*Completions*' even without user input.

> > - If there are a lot of completions, the completions
> >   buffer can take a while to update...

These options help with that:

3. `icicle-incremental-completion-threshold':

   When there are more than this many completions then
   wait (see next) before updating *Completions*.

4. `icicle-incremental-completion-delay':

   Seconds to wait before updating *Completions*, if
   there are currently more matching candidates than
   `icicle-incremental-completion-threshold'.

5. `icicle-max-candidates':

   Non-nil means show at most this many candidates
   in *Completions*.
___

https://www.emacswiki.org/emacs/Icicles_-_Icompletion#IncrementalCompletion

Re: [External] : Re: Updating *Completions* as you type

[Philip Kaludercic]

Drew Adams <drew.adams@oracle.com> writes:

> FWIW:
>
> I'll mention some similar features in Icicles.  They
> were introduced together from the outset (in 2005!),
> and have proven useful.

You have mentioned Icicles a number of times, and one of the main
reasons I have never tried it out is that it is not available on ELPA or
any kind of official repository.  I believe wrote about this a while
back, but according to [0], the main issue remains that your libraries
overwrite Emacs primitives, as you mention in the main file:

--8<---------------cut here---------------start------------->8---
;;  ***** NOTE: These EMACS PRIMITIVES have been REDEFINED in Icicles:
;;
;;  `completing-read'              - (See below and doc string.)
;;  `display-completion-list'      - (See below and doc string.)
;;  `exit-minibuffer'              - Remove *Completion* window.
;;  `minibuffer-complete-and-exit' - Remove *Completion* window.
;;  `read-file-name'               - (See below and doc string.)
;;  `read-from-minibuffer'         - (See below and doc string.)
;;  `read-string'                  - (See below and doc string.)
;;
;;
;;  ***** NOTE: The following functions defined in `dabbrev.el' have
;;              been REDEFINED in Icicles:
;;
;;  `dabbrev-completion' - Use Icicles completion when you repeat
;;                         (`C-M-/').
;;
;;
;;  ***** NOTE: The following functions defined in `lisp.el' have
;;              been REDEFINED in Icicles:
;;
;;  `lisp-complete-symbol' - Selects `*Completions*' window even if on
;;                           another frame.
;;
;;
;;  ***** NOTE: The following functions defined in `mouse.el' have
;;              been REDEFINED in Icicles:
;;
;;  `mouse-choose-completion' - Return the number of the completion.
;;
;;
;;  ***** NOTE: The following functions defined in `simple.el' have
;;              been REDEFINED in Icicles:
;;
;;  `choose-completion-string' -
;;     Don't exit minibuffer after `lisp-complete-symbol' completion.
;;  `completion-setup-function' - 1. Put faces on inserted string(s).
;;                                2. Help on help.
;;  `switch-to-completions' - Always selects `*Completions*' window.
;;
;;  `next-history-element' (advised only) -
;;     Depending on `icicle-default-value', select minibuffer
;;     contents.
;;
;;  `repeat-complex-command' - Use `completing-read' to read command.
;;
;;  For descriptions of changes to this file, see `icicles-chg.el'.
--8<---------------cut here---------------end--------------->8---

Is this really necessary?  Somehow all the other completion systems
appear to get by without these kinds of invasive changes, and it would
be a pity the barriers for trying out your package continue to be
unnecessarily high, if the functionality is so advanced.  At the very
least it would be nice if you could have your own ELPA, perhaps hosted
on Emacs Wiki to distribute these packages in a more standardised way.

[0] https://www.emacswiki.org/emacs/icicles.el

-- 
Philip Kaludercic

RE: [External] : Re: Updating *Completions* as you type

[Drew Adams]

I'm not asking you, or anyone, to use Icicles.
Unfortunately, talking about its features can
perhaps give the opposite impression.

As I tried to make clear in my posts, their aim
was to mention various Icicles features I think
are relevant to this thread - not to advertise
Icicles or suggest its use.

The point was to encourage consideration of such
or similar features as useful and perhaps worth
adding to vanilla Emacs.  Nothing more.  

Icicles is just a reference point here.  Its doc
about such features might hopefully provide some
food for thought.  _What users can do_ is the
point, not how such features can be implemented.

It's good to see others coming around to similar
feature ideas now.  Knowledge that such features
work _in combination_, and they have done so for
quite a while, should be helpful, I hope.
___

Wrt la petite histoire -

To my knowledge, the first appearance of any
incremental completion was in icomplete-mode.
But that wasn't a completion whose result you
could _use_; it was only completion you could
_see_ (and only a few completions).  You could
use it only as a guide/preview of what you could
then type.  (Much later, completion of input was
finally added to icomplete-mode.)

Next was IswitchB, which later became the model
for Ido.  Completion candidates were shown only
in the minibuffer (and again, only a few) - no
use of *Completions* to show candidates.

Icicles came after icomplete-mode and IswitchB,
and before Ido.  It introduced incremental
completion showing candidates in *Completions*;
cycling among candidates (notion of "current"
candidate); on-the-fly sorting of them; etc.

Re: [External] : Re: Updating *Completions* as you type

[Philip Kaludercic]

Drew Adams <drew.adams@oracle.com> writes:

> I'm not asking you, or anyone, to use Icicles.
> Unfortunately, talking about its features can
> perhaps give the opposite impression.

I didn't understand it that way either, my I just wanted to point out
that my impressions is that people miss out on even trying your packages
out, because they aren't distributed on any standard archive.  The
reason they aren't distributed on GNU or NonGNU ELPA is that you
overwrite built-in definitions -- hence my question: Are you sure this
is necessary, since if not and you could change them, your packages
could be added to GNU/NonGNU ELPA and more people could profit from your
work.

> As I tried to make clear in my posts, their aim
> was to mention various Icicles features I think
> are relevant to this thread - not to advertise
> Icicles or suggest its use.
>
> The point was to encourage consideration of such
> or similar features as useful and perhaps worth
> adding to vanilla Emacs.  Nothing more.  
>
> Icicles is just a reference point here.  Its doc
> about such features might hopefully provide some
> food for thought.  _What users can do_ is the
> point, not how such features can be implemented.
>
> It's good to see others coming around to similar
> feature ideas now.  Knowledge that such features
> work _in combination_, and they have done so for
> quite a while, should be helpful, I hope.

I might be mistaken, but the reason I thought your were "advertising"
Icicles, was that you usually respond to the suggestion to add some
feature with something like "Icicles had this feature since 200X".

> ___
>
> Wrt la petite histoire -
>
> To my knowledge, the first appearance of any
> incremental completion was in icomplete-mode.
> But that wasn't a completion whose result you
> could _use_; it was only completion you could
> _see_ (and only a few completions).  You could
> use it only as a guide/preview of what you could
> then type.  (Much later, completion of input was
> finally added to icomplete-mode.)
>
> Next was IswitchB, which later became the model
> for Ido.  Completion candidates were shown only
> in the minibuffer (and again, only a few) - no
> use of *Completions* to show candidates.
>
> Icicles came after icomplete-mode and IswitchB,
> and before Ido.  It introduced incremental
> completion showing candidates in *Completions*;
> cycling among candidates (notion of "current"
> candidate); on-the-fly sorting of them; etc.

The copyright files indicate this chronology:

icomplete: 1992
icicles:   1995
iswitchb:  1996
ido:       1996

RE: [External] : Re: Updating *Completions* as you type

[Drew Adams]

> The reason they aren't distributed on GNU or
> NonGNU ELPA is that you overwrite built-in
> definitions

No, that's not the reason.  Not at all.

> > As I tried to make clear in my posts, their aim
> > was to mention various Icicles features I think
> > are relevant to this thread - not to advertise
> > Icicles or suggest its use.
> >
> > The point was to encourage consideration of such
> > or similar features as useful and perhaps worth
> > adding to vanilla Emacs.  Nothing more.
> >
> > Icicles is just a reference point here.  Its doc
> > about such features might hopefully provide some
> > food for thought.  _What users can do_ is the
> > point, not how such features can be implemented.
> >
> > It's good to see others coming around to similar
> > feature ideas now.  Knowledge that such features
> > work _in combination_, and they have done so for
> > quite a while, should be helpful, I hope.
> 
> I might be mistaken, but the reason I thought your
> were "advertising" Icicles, was that you usually
> respond to the suggestion to add some feature with
> something like "Icicles had this feature since 200X".

I don't know whether you're mistaken about
why you thought that.

But if what you mean is that you might be
mistaken about why I say that, then yes,
you're apparently mistaken about that.

The above text should make that "why" very
clear, so I wonder why you would still be
wondering.  Let me say it once more:

  such features might hopefully provide
  some food for thought.  _What users can
  do_ is the point

Many Icicles features have been adopted
(reinvented) for vanilla Emacs or other
completion libraries.  Other features
have not (yet).  I mentioned on-the-fly,
flexible interactive sorting as one, in
particular, because this thread broached
the question of sorting candidates.

The point in mentioning Icicles features
is that they might be of interest as
food for thought: _what_ they do, not
necessarily _how_ they do it.

If a user of vanilla Emacs or another
library can't do XYZ, and a user of
Icicles can, and if XYZ is a useful
thing to be able to do, then someone
might want to look at the feature in
Icicles, as food for thought.

It's not important that Icicles does
something, except that from Icicles you
can get a good description of a feature
and see the source code - if interested.

> > Wrt la petite histoire -
> >
> > To my knowledge, the first appearance of any
> > incremental completion was in icomplete-mode.
> > But that wasn't a completion whose result you
> > could _use_; it was only completion you could
> > _see_ (and only a few completions).  You could
> > use it only as a guide/preview of what you could
> > then type.  (Much later, completion of input was
> > finally added to icomplete-mode.)
> >
> > Next was IswitchB, which later became the model
> > for Ido.  Completion candidates were shown only
> > in the minibuffer (and again, only a few) - no
> > use of *Completions* to show candidates.
> >
> > Icicles came after icomplete-mode and IswitchB,
> > and before Ido.  It introduced incremental
> > completion showing candidates in *Completions*;
> > cycling among candidates (notion of "current"
> > candidate); on-the-fly sorting of them; etc.
> 
> The copyright files indicate this chronology:
> 
> icomplete: 1992
> icicles:   1995
> iswitchb:  1996
> ido:       1996

Please read what I said, which is about
incremental completion, not copyright of
the libraries.

Here's a very brief description of the
initial history of Icicles:

https://www.emacswiki.org/emacs/Icicles_-_La_Petite_Histoire

That doesn't specifically mention
incremental completion.  But the Change
Log in icicles-chg.el indicates it was
added in 2006 or before then.  Looking
at details in the log indicates it was
in 2005 or before (as "icompletion").

Icicles as such, with cycling of
matching candidates etc., really began
in 2005.  (The name dates from then.)

From 1996 to 2005 it was essentially
elect-mbuf.el, which (1) never updated
the set of candidates to cycle among,
and (2) never tied cycling (current
candidate in *Completions*) to the
minibuffer content (no "current
candidate" there).  It didn't do much
at all, except let you see the initial
set of completion candidates.

[ BTW, #2 is similar to the situation
  with vanilla Emacs today wrt sorting:
  the sort order for *Completions*
  isn't necessarily the same as the
  sort order for the minibuffer content
  (`display-sort-function' versus
  `cycle-sort-function').

  I've asked for the reason why - why
  or when that might be useful, but
  I've not received any answer, ever.
  Until I do, I'll just consider it a
  misfeature or a missed opportunity. ]

Re: [External] : Re: Updating *Completions* as you type

[Emanuel Berg]

Philip Kaludercic wrote:

> You have mentioned Icicles a number of times, and one of the
> main reasons I have never tried it out is that it is not
> available on ELPA or any kind of official repository.
> I believe wrote about this a while back, but according to
> [...], the main issue remains that your libraries overwrite
> Emacs primitives, as you mention in the main file:
>
> ;;  ***** NOTE: These EMACS PRIMITIVES have been REDEFINED in Icicles:
> ;;
> ;;  `completing-read'              - (See below and doc string.)
> ;;  `display-completion-list'      - (See below and doc string.)
> ;;  `exit-minibuffer'              - Remove *Completion* window.
> ;;  `minibuffer-complete-and-exit' - Remove *Completion* window.
> ;;  `read-file-name'               - (See below and doc string.)
> ;;  `read-from-minibuffer'         - (See below and doc string.)
> ;;  `read-string'                  - (See below and doc string.)

Meta-Note: If "Emacs primitive" equals a "built-in" function
equals a function written in C, then
`display-completion-list', `exit-minibuffer',
`minibuffer-complete-and-exit', and `read-file-name' are
actually ordinary Lisp functions, written in Lisp.

Is the reason Icicles is not a package in GNU ELPA that it
redefines functions? If so, see if one can reduce the number
of such cases one by one.

I say this because I have some experience from similar
situations where you feel there is a whole bunch of things to
do, and it feels impossible, almost. After removing one such
case, it still feels that way. But after removing two, it
already feels much better and after removing three, in terms
of how you mentally experience the task, it is done, almost.

-- 
underground experts united
https://dataswamp.org/~incal

Re: [External] : Re: Updating *Completions* as you type

[Emanuel Berg]

>> ;;  ***** NOTE: These EMACS PRIMITIVES have been REDEFINED in Icicles:
>> ;;
>> ;;  `completing-read'              - (See below and doc string.)
>> ;;  `display-completion-list'      - (See below and doc string.)
>> ;;  `exit-minibuffer'              - Remove *Completion* window.
>> ;;  `minibuffer-complete-and-exit' - Remove *Completion* window.
>> ;;  `read-file-name'               - (See below and doc string.)
>> ;;  `read-from-minibuffer'         - (See below and doc string.)
>> ;;  `read-string'                  - (See below and doc string.)
>
> Meta-Note: If "Emacs primitive" equals a "built-in" function
> equals a function written in C, then
> `display-completion-list', `exit-minibuffer',
> `minibuffer-complete-and-exit', and `read-file-name' are
> actually ordinary Lisp functions, written in Lisp.

And all those are in minibuffer.el.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Updating *Completions* as you type

[Michael Heerdegen]

sbaugh@catern.com writes:

> It would be nice if there was a built-in customization which caused
> *Completions* to update as you type, as long as that buffer is visible.

AFAIU this sounds very much like "aggressive-completion" in Gnu Elpa.

Michael.