Re: [PATCH] `completing-read`: Add `group-function` support to completion metadata (REVISED PATCH VERSION 7)

[Daniel Mendler <mail@daniel-mendler.de>]

[-- Attachment #1: Type: text/plain, Size: 599 bytes --]

On 5/13/21 12:32 PM, Eli Zaretskii wrote:
>> I attached the updated patches, the variable `completions-group-sort`
>> allows the more user friendly values nil and 'alphabetical. I hope it is
>> more acceptable in this form.
> 
> Thanks, this addresses my concerns.
> 
> (Please proofread the comments, doc strings, and Texinfo changes in
> the patches to make sure you leave 2 spaces between sentences, per our
> conventions to use US English.)

Thank you, Eli. I went over the commit messages, comments, Texinfo and
doc strings and made a few corrections. The revised patches are attached.

Daniel

[-- Attachment #2: 0001-completing-read-Add-group-function-to-the-completion.patch --]
[-- Type: text/x-diff, Size: 18741 bytes --]

From 0fc290d79e9dbb763517d61eb3e16c3cd2a108d7 Mon Sep 17 00:00:00 2001
From: Daniel Mendler <mail@daniel-mendler.de>
Date: Sun, 25 Apr 2021 13:07:29 +0200
Subject: [PATCH 1/5] (completing-read): Add `group-function` to the completion
 metadata

A completion table can specify a `group-function` in its metadata.
The group function takes two arguments, a completion candidate and a
transform argument.  The group function is used to group the
candidates after sorting and to enhance the completion UI with group
titles.

If the transform argument is nil, the function must return the title
of the group to which the completion candidate belongs.  The function
may also return nil if the candidate does not belong to a group.

if the transform argument is non-nil, the function must return the
transformed candidate.  For example, the transformation allows to
remove a redundant part of the candidate, which is then displayed in
the title.

The grouping functionality is guarded by the customizable variable
`completions-group` and turned off by default for the *Completions*
buffer.

The specific form of the `group-function` has been chosen in order to
allow allocation-free grouping.  This is important for completion UIs,
which continously update the displayed set of candidates (Icomplete,
Vertico, Ivy, etc.).  Only when the transform argument is non-nil the
candidate transformation is performed, which may involve a string
allocation as done in the function `xref--completing-read-group`.

The function `xref-show-definitions-completing-read` makes use of the
`group-function`, by moving the file name prefix to the title.  If
grouping is enabled, the *Completions* are displayed as
"linenum:summary" instead of "file:linenum:summary".  This way the
*Completions* buffer resembles the *Occur* buffer.

* doc/lispref/minibuf.texi: Add documentation.

* lisp/minibuffer.el (completion-metadata): Describe the
`group-function` in the docstring.
(completions-group): Add guard variable, off by default.
(completions-group-format): Add variable defining the format string
for the group titles.
(completions-group-title): Add face used by `completions-group-format`
for the group titles.
(completions-group-separator): Add face used by
`completions-group-format` for the group separator lines.
(minibuffer--group-by): New grouping function.
(minibuffer-completion-help): Use it.
(display-completion-list): Add optional GROUP-FUN argument.
(completion--insert-strings): Add optional GROUP-FUN argument.  Insert
group titles if `completions-format` is `one-column`.  Transform each
candidate with the GROUP-FUN.  Attach the untransformed candidate to
the property `completion--string`.

* lisp/simple.el (choose-completion): Retrieve the untransformed
completion candidate from the property `completion--string`.

* lisp/progmodes/xref.el:
(xref--completing-read-group): New grouping function.
(xref-show-definitions-completing-read): Use it.
---
 doc/lispref/minibuf.texi |  10 ++++
 lisp/minibuffer.el       | 126 ++++++++++++++++++++++++++++++++-------
 lisp/progmodes/xref.el   |  18 ++++--
 lisp/simple.el           |  11 ++--
 4 files changed, 134 insertions(+), 31 deletions(-)

diff --git a/doc/lispref/minibuf.texi b/doc/lispref/minibuf.texi
index 145eee8f06..196dd99076 100644
--- a/doc/lispref/minibuf.texi
+++ b/doc/lispref/minibuf.texi
@@ -1943,6 +1943,16 @@ Programmed Completion
 a suffix displayed after the completion string.  This function
 takes priority over @code{annotation-function}.
 
+@item group-function
+The value should be a function for grouping the completion candidates.
+The function must take two arguments, @var{completion}, which is a
+completion candidate and @var{transform}, which is a boolean flag.  If
+@var{transform} is @code{nil}, the function must return the group
+title of the group to which the candidate belongs.  The returned title
+can also be @code{nil}.  Otherwise the function must return the
+transformed candidate.  The transformation can for example remove a
+redundant prefix, which is displayed in the group title.
+
 @item display-sort-function
 The value should be a function for sorting completions.  The function
 should take one argument, a list of completion strings, and return a
diff --git a/lisp/minibuffer.el b/lisp/minibuffer.el
index caf06ec710..b5245e239e 100644
--- a/lisp/minibuffer.el
+++ b/lisp/minibuffer.el
@@ -126,6 +126,13 @@ completion-metadata
    three-element lists: completion, its prefix and suffix.  This
    function takes priority over `annotation-function' when both are
    provided, so only this function is used.
+- `group-function': function for grouping the completion candidates.
+   Takes two arguments: a completion candidate (COMPLETION) and a
+   boolean flag (TRANSFORM).  If TRANSFORM is nil, the function
+   returns the group title of the group to which the candidate
+   belongs.  The returned title may be nil.  Otherwise the function
+   returns the transformed candidate.  The transformation can remove a
+   redundant prefix, which is displayed in the group title.
 - `display-sort-function': function to sort entries in *Completions*.
    Takes one argument (COMPLETIONS) and should return a new list
    of completions.  Can operate destructively.
@@ -1138,6 +1145,32 @@ completion-cycle-threshold
   :version "24.1"
   :type completion--cycling-threshold-type)
 
+(defcustom completions-group nil
+  "Enable grouping of completion candidates in the *Completions* buffer.
+See also `completions-group-format'."
+  :type 'boolean
+  :version "28.1")
+
+(defcustom completions-group-format
+  (concat
+   (propertize "    " 'face 'completions-group-separator)
+   (propertize " %s " 'face 'completions-group-title)
+   (propertize " " 'face 'completions-group-separator
+               'display '(space :align-to right)))
+  "Format string used for the group title."
+  :type 'string
+  :version "28.1")
+
+(defface completions-group-title
+  '((t :inherit shadow :slant italic))
+  "Face used for the title text of the candidate group headlines."
+  :version "28.1")
+
+(defface completions-group-separator
+  '((t :inherit shadow :strike-through t))
+  "Face used for the separator lines between the candidate groups."
+  :version "28.1")
+
 (defun completion--cycle-threshold (metadata)
   (let* ((cat (completion-metadata-get metadata 'category))
          (over (completion--category-override cat 'cycle)))
@@ -1401,6 +1434,17 @@ minibuffer--sort-preprocess-history
                      (substring c base-size)))
                  hist)))))
 
+(defun minibuffer--group-by (fun elems)
+  "Group ELEMS by FUN."
+  (let ((groups))
+    (dolist (cand elems)
+      (let* ((key (funcall fun cand nil))
+             (group (assoc key groups)))
+        (if group
+            (setcdr group (cons cand (cdr group)))
+          (push (list key cand) groups))))
+    (mapcan (lambda (x) (nreverse (cdr x))) (nreverse groups))))
+
 (defun completion-all-sorted-completions (&optional start end)
   (or completion-all-sorted-completions
       (let* ((start (or start (minibuffer-prompt-end)))
@@ -1747,11 +1791,17 @@ completions-detailed
   :type 'boolean
   :version "28.1")
 
-(defun completion--insert-strings (strings)
+;; TODO: Split up this function in one function per `completions-format'.
+;; TODO: Add group title support for horizontal and vertical format.
+(defun completion--insert-strings (strings &optional group-fun)
   "Insert a list of STRINGS into the current buffer.
-Uses columns to keep the listing readable but compact.
-It also eliminates runs of equal strings."
+Uses columns to keep the listing readable but compact.  It also
+eliminates runs of equal strings.  GROUP-FUN is a `group-function'
+used for grouping the completion."
   (when (consp strings)
+    ;; FIXME: Currently grouping is enabled only for the 'one-column format.
+    (unless (eq completions-format 'one-column)
+      (setq group-fun nil))
     (let* ((length (apply #'max
 			  (mapcar (lambda (s)
 				    (if (consp s)
@@ -1768,6 +1818,7 @@ completion--insert-strings
 		     (max 1 (/ (length strings) 2))))
 	   (colwidth (/ wwidth columns))
            (column 0)
+           (last-title nil)
 	   (rows (/ (length strings) columns))
 	   (row 0)
            (first t)
@@ -1780,6 +1831,13 @@ completion--insert-strings
       ;; The insertion should be "sensible" no matter what choices were made
       ;; for the parameters above.
       (dolist (str strings)
+        ;; Add group titles.
+        (when group-fun
+          (let ((title (funcall group-fun (if (consp str) (car str) str) nil)))
+            (unless (equal title last-title)
+              (when title
+                (insert (format completions-group-format title) "\n"))
+              (setq last-title title))))
 	(unless (equal laststring str) ; Remove (consecutive) duplicates.
 	  (setq laststring str)
           ;; FIXME: `string-width' doesn't pay attention to
@@ -1825,8 +1883,15 @@ completion--insert-strings
 		  nil))))
             (setq first nil)
             (if (not (consp str))
-                (put-text-property (point) (progn (insert str) (point))
-                                   'mouse-face 'highlight)
+                (add-text-properties
+                 (point)
+                 (progn
+                   (insert
+                    (if group-fun
+                        (funcall group-fun str 'transform)
+                      str))
+                   (point))
+                 `(mouse-face highlight completion--string ,str))
               ;; If `str' is a list that has 2 elements,
               ;; then the second element is a suffix annotation.
               ;; If `str' has 3 elements, then the second element
@@ -1837,8 +1902,15 @@ completion--insert-strings
                   (let ((beg (point))
                         (end (progn (insert prefix) (point))))
                     (put-text-property beg end 'mouse-face nil)))
-                (put-text-property (point) (progn (insert (car str)) (point))
-                                   'mouse-face 'highlight)
+                (add-text-properties
+                 (point)
+                 (progn
+                   (insert
+                    (if group-fun
+                         (funcall group-fun (car str) 'transform)
+                      (car str)))
+                   (point))
+                 `(mouse-face highlight completion--string ,(car str)))
                 (let ((beg (point))
                       (end (progn (insert suffix) (point))))
                   (put-text-property beg end 'mouse-face nil)
@@ -1923,7 +1995,7 @@ completion-hilit-commonality
         completions)
        base-size))))
 
-(defun display-completion-list (completions &optional common-substring)
+(defun display-completion-list (completions &optional common-substring group-fun)
   "Display the list of completions, COMPLETIONS, using `standard-output'.
 Each element may be just a symbol or string
 or may be a list of two strings to be printed as if concatenated.
@@ -1933,7 +2005,9 @@ display-completion-list
 The actual completion alternatives, as inserted, are given `mouse-face'
 properties of `highlight'.
 At the end, this runs the normal hook `completion-setup-hook'.
-It can find the completion buffer in `standard-output'."
+It can find the completion buffer in `standard-output'.
+GROUP-FUN is a `group-function' used for grouping the completion
+candidates."
   (declare (advertised-calling-convention (completions) "24.4"))
   (if common-substring
       (setq completions (completion-hilit-commonality
@@ -1946,7 +2020,7 @@ display-completion-list
 	(let ((standard-output (current-buffer))
 	      (completion-setup-hook nil))
           (with-suppressed-warnings ((callargs display-completion-list))
-	    (display-completion-list completions common-substring)))
+	    (display-completion-list completions common-substring group-fun)))
 	(princ (buffer-string)))
 
     (with-current-buffer standard-output
@@ -1954,7 +2028,7 @@ display-completion-list
       (if (null completions)
           (insert "There are no possible completions of what you have typed.")
         (insert "Possible completions are:\n")
-        (completion--insert-strings completions))))
+        (completion--insert-strings completions group-fun))))
 
   (run-hooks 'completion-setup-hook)
   nil)
@@ -2067,6 +2141,9 @@ minibuffer-completion-help
              (aff-fun (or (completion-metadata-get all-md 'affixation-function)
                           (plist-get completion-extra-properties
                                      :affixation-function)))
+             (sort-fun (completion-metadata-get all-md 'display-sort-function))
+             (group-fun (and completions-group
+                             (completion-metadata-get all-md 'group-function)))
              (mainbuf (current-buffer))
              ;; If the *Completions* buffer is shown in a new
              ;; window, mark it as softly-dedicated, so bury-buffer in
@@ -2098,15 +2175,22 @@ minibuffer-completion-help
                       ;; Remove the base-size tail because `sort' requires a properly
                       ;; nil-terminated list.
                       (when last (setcdr last nil))
-                      (setq completions
-                            ;; FIXME: This function is for the output of all-completions,
-                            ;; not completion-all-completions.  Often it's the same, but
-                            ;; not always.
-                            (let ((sort-fun (completion-metadata-get
-                                             all-md 'display-sort-function)))
-                              (if sort-fun
-                                  (funcall sort-fun completions)
-                                (sort completions 'string-lessp))))
+
+                      ;; Sort first using the `display-sort-function'.
+                      ;; FIXME: This function is for the output of
+                      ;; all-completions, not
+                      ;; completion-all-completions.  Often it's the
+                      ;; same, but not always.
+                      (setq completions (if sort-fun
+                                            (funcall sort-fun completions)
+                                          (sort completions 'string-lessp)))
+
+                      ;; After sorting, group the candidates using the
+                      ;; `group-function'.
+                      (when group-fun
+                        (setq completions
+                              (minibuffer--group-by group-fun completions)))
+
                       (cond
                        (aff-fun
                         (setq completions
@@ -2152,7 +2236,7 @@ minibuffer-completion-help
                                                      (if (eq (car bounds) (length result))
                                                          'exact 'finished)))))))
 
-                      (display-completion-list completions)))))
+                      (display-completion-list completions nil group-fun)))))
           nil)))
     nil))
 
diff --git a/lisp/progmodes/xref.el b/lisp/progmodes/xref.el
index 7fc7181acc..2a4fb2c417 100644
--- a/lisp/progmodes/xref.el
+++ b/lisp/progmodes/xref.el
@@ -1044,6 +1044,12 @@ xref-show-definitions-buffer-at-bottom
 (define-obsolete-function-alias 'xref--show-defs-buffer-at-bottom
   #'xref-show-definitions-buffer-at-bottom "28.1")
 
+(defun xref--completing-read-group (cand transform)
+  "Return group title of candidate CAND or TRANSFORM the candidate."
+  (if transform
+      (substring cand (1+ (next-single-property-change 0 'xref--group cand)))
+    (get-text-property 0 'xref--group cand)))
+
 (defun xref-show-definitions-completing-read (fetcher alist)
   "Let the user choose the target definition with completion.
 
@@ -1072,10 +1078,12 @@ xref-show-definitions-completing-read
                                     (format #("%d:" 0 2 (face xref-line-number))
                                             line)
                                   ""))
+                               (group-prefix
+                                (substring group group-prefix-length))
                                (group-fmt
-                                (propertize
-                                 (substring group group-prefix-length)
-                                 'face 'xref-file-header))
+                                (propertize group-prefix
+                                            'face 'xref-file-header
+                                            'xref--group group-prefix))
                                (candidate
                                 (format "%s:%s%s" group-fmt line-fmt summary)))
                           (push (cons candidate xref) xref-alist-with-line-info)))))
@@ -1087,7 +1095,9 @@ xref-show-definitions-completing-read
                          (lambda (string pred action)
                            (cond
                             ((eq action 'metadata)
-                             '(metadata . ((category . xref-location))))
+                             `(metadata
+                               . ((category . xref-location)
+                                  (group-function . ,#'xref--completing-read-group))))
                             (t
                              (complete-with-action action collection string pred)))))
                         (def (caar collection)))
diff --git a/lisp/simple.el b/lisp/simple.el
index 35bb472be0..db8e26a2c0 100644
--- a/lisp/simple.el
+++ b/lisp/simple.el
@@ -8893,18 +8893,17 @@ choose-completion
           (choice
            (save-excursion
              (goto-char (posn-point (event-start event)))
-             (let (beg end)
+             (let (beg)
                (cond
                 ((and (not (eobp)) (get-text-property (point) 'mouse-face))
-                 (setq end (point) beg (1+ (point))))
+                 (setq beg (1+ (point))))
                 ((and (not (bobp))
                       (get-text-property (1- (point)) 'mouse-face))
-                 (setq end (1- (point)) beg (point)))
+                 (setq beg (point)))
                 (t (error "No completion here")))
                (setq beg (previous-single-property-change beg 'mouse-face))
-               (setq end (or (next-single-property-change end 'mouse-face)
-                             (point-max)))
-               (buffer-substring-no-properties beg end)))))
+               (substring-no-properties
+                (get-text-property beg 'completion--string))))))
 
       (unless (buffer-live-p buffer)
         (error "Destination buffer is dead"))
-- 
2.20.1


[-- Attachment #3: 0002-completion-insert-strings-Split-function-Full-group-.patch --]
[-- Type: text/x-diff, Size: 13753 bytes --]

From 5a55152b4064ea674fea055a8c48a0824b70213d Mon Sep 17 00:00:00 2001
From: Daniel Mendler <mail@daniel-mendler.de>
Date: Fri, 30 Apr 2021 08:40:59 +0200
Subject: [PATCH 2/5] (completion--insert-strings): Split function; Full group
 title support

Split `completion--insert-strings` into a function per completions
format in order to increase readability and extensibility.  This
change eases the addition of more formats.  Add support for group
titles to the vertical and horizontal formatting functions.

* minibuffer.el (completion--insert): Add new function.
(completion--insert-vertical, completion--insert-horizontal,
completion--insert-one-column): Extract function from
`completion--insert-strings`.  Use new function `completion--insert`.
(completion--insert-strings): Use new insertion functions.
---
 lisp/minibuffer.el | 276 ++++++++++++++++++++++++++-------------------
 1 file changed, 158 insertions(+), 118 deletions(-)

diff --git a/lisp/minibuffer.el b/lisp/minibuffer.el
index b5245e239e..8b3f332782 100644
--- a/lisp/minibuffer.el
+++ b/lisp/minibuffer.el
@@ -1791,21 +1791,17 @@ completions-detailed
   :type 'boolean
   :version "28.1")
 
-;; TODO: Split up this function in one function per `completions-format'.
-;; TODO: Add group title support for horizontal and vertical format.
 (defun completion--insert-strings (strings &optional group-fun)
   "Insert a list of STRINGS into the current buffer.
-Uses columns to keep the listing readable but compact.  It also
-eliminates runs of equal strings.  GROUP-FUN is a `group-function'
-used for grouping the completion."
+The candidate strings are inserted into the buffer depending on the
+completions format as specified by the variable `completions-format'.
+Runs of equal candidate strings are eliminated.  GROUP-FUN is a
+`group-function' used for grouping the completion candidates."
   (when (consp strings)
-    ;; FIXME: Currently grouping is enabled only for the 'one-column format.
-    (unless (eq completions-format 'one-column)
-      (setq group-fun nil))
     (let* ((length (apply #'max
 			  (mapcar (lambda (s)
 				    (if (consp s)
-					(apply #'+ (mapcar #'string-width s))
+				        (apply #'+ (mapcar #'string-width s))
 				      (string-width s)))
 				  strings)))
 	   (window (get-buffer-window (current-buffer) 0))
@@ -1816,126 +1812,170 @@ completion--insert-strings
 		     ;; Don't allocate more columns than we can fill.
 		     ;; Windows can't show less than 3 lines anyway.
 		     (max 1 (/ (length strings) 2))))
-	   (colwidth (/ wwidth columns))
-           (column 0)
-           (last-title nil)
-	   (rows (/ (length strings) columns))
-	   (row 0)
-           (first t)
-	   (laststring nil))
+	   (colwidth (/ wwidth columns)))
       (unless (or tab-stop-list (null completion-tab-width)
                   (zerop (mod colwidth completion-tab-width)))
         ;; Align to tab positions for the case
         ;; when the caller uses tabs inside prefix.
         (setq colwidth (- colwidth (mod colwidth completion-tab-width))))
-      ;; The insertion should be "sensible" no matter what choices were made
-      ;; for the parameters above.
-      (dolist (str strings)
-        ;; Add group titles.
+      (funcall (intern (format "completion--insert-%s" completions-format))
+               strings group-fun length wwidth colwidth columns))))
+
+(defun completion--insert-horizontal (strings group-fun
+                                              length wwidth
+                                              colwidth _columns)
+  (let ((column 0)
+        (first t)
+	(last-title nil)
+        (last-string nil))
+    (dolist (str strings)
+      (unless (equal last-string str) ; Remove (consecutive) duplicates.
+	(setq last-string str)
         (when group-fun
           (let ((title (funcall group-fun (if (consp str) (car str) str) nil)))
             (unless (equal title last-title)
+              (setq last-title title)
               (when title
-                (insert (format completions-group-format title) "\n"))
-              (setq last-title title))))
-	(unless (equal laststring str) ; Remove (consecutive) duplicates.
-	  (setq laststring str)
+                (insert (if first "" "\n") (format completions-group-format title) "\n")
+                (setq column 0
+                      first t)))))
+	(unless first
           ;; FIXME: `string-width' doesn't pay attention to
           ;; `display' properties.
-          (let ((length (if (consp str)
-                            (apply #'+ (mapcar #'string-width str))
-                          (string-width str))))
-            (cond
-             ((eq completions-format 'one-column)
-              ;; Nothing special
-              )
-	     ((eq completions-format 'vertical)
-	      ;; Vertical format
-	      (when (> row rows)
-		(forward-line (- -1 rows))
-		(setq row 0 column (+ column colwidth)))
-	      (when (> column 0)
-		(end-of-line)
-		(while (> (current-column) column)
-		  (if (eobp)
-		      (insert "\n")
-		    (forward-line 1)
-		    (end-of-line)))
-		(insert " \t")
-		(set-text-properties (1- (point)) (point)
-				     `(display (space :align-to ,column)))))
-	     (t
-	      ;; Horizontal format
-	      (unless first
-		(if (< wwidth (+ (max colwidth length) column))
-		    ;; No space for `str' at point, move to next line.
-		    (progn (insert "\n") (setq column 0))
-		  (insert " \t")
-		  ;; Leave the space unpropertized so that in the case we're
-		  ;; already past the goal column, there is still
-		  ;; a space displayed.
-		  (set-text-properties (1- (point)) (point)
-				       ;; We can set tab-width using
-				       ;; completion-tab-width, but
-				       ;; the caller can prefer using
-				       ;; \t to align prefixes.
-				       `(display (space :align-to ,column)))
-		  nil))))
-            (setq first nil)
-            (if (not (consp str))
-                (add-text-properties
-                 (point)
-                 (progn
-                   (insert
-                    (if group-fun
-                        (funcall group-fun str 'transform)
-                      str))
-                   (point))
-                 `(mouse-face highlight completion--string ,str))
-              ;; If `str' is a list that has 2 elements,
-              ;; then the second element is a suffix annotation.
-              ;; If `str' has 3 elements, then the second element
-              ;; is a prefix, and the third element is a suffix.
-              (let* ((prefix (when (nth 2 str) (nth 1 str)))
-                     (suffix (or (nth 2 str) (nth 1 str))))
-                (when prefix
-                  (let ((beg (point))
-                        (end (progn (insert prefix) (point))))
-                    (put-text-property beg end 'mouse-face nil)))
-                (add-text-properties
-                 (point)
-                 (progn
-                   (insert
-                    (if group-fun
-                         (funcall group-fun (car str) 'transform)
-                      (car str)))
-                   (point))
-                 `(mouse-face highlight completion--string ,(car str)))
-                (let ((beg (point))
-                      (end (progn (insert suffix) (point))))
-                  (put-text-property beg end 'mouse-face nil)
-                  ;; Put the predefined face only when suffix
-                  ;; is added via annotation-function without prefix,
-                  ;; and when the caller doesn't use own face.
-                  (unless (or prefix (text-property-not-all
-                                      0 (length suffix) 'face nil suffix))
-                    (font-lock-prepend-text-property
-                     beg end 'face 'completions-annotations)))))
-	    (cond
-             ((eq completions-format 'one-column)
-              (insert "\n"))
-	     ((eq completions-format 'vertical)
-	      ;; Vertical format
-	      (if (> column 0)
-		  (forward-line)
-		(insert "\n"))
-	      (setq row (1+ row)))
-	     (t
-	      ;; Horizontal format
-	      ;; Next column to align to.
-	      (setq column (+ column
-			      ;; Round up to a whole number of columns.
-			      (* colwidth (ceiling length colwidth))))))))))))
+	  (if (< wwidth (+ column (max colwidth
+                                       (if (consp str)
+                                           (apply #'+ (mapcar #'string-width str))
+                                         (string-width str)))))
+	      ;; No space for `str' at point, move to next line.
+	      (progn (insert "\n") (setq column 0))
+	    (insert " \t")
+	    ;; Leave the space unpropertized so that in the case we're
+	    ;; already past the goal column, there is still
+	    ;; a space displayed.
+	    (set-text-properties (1- (point)) (point)
+				 ;; We can set tab-width using
+				 ;; completion-tab-width, but
+				 ;; the caller can prefer using
+				 ;; \t to align prefixes.
+				 `(display (space :align-to ,column)))
+	    nil))
+        (setq first nil)
+        (completion--insert str group-fun)
+	;; Next column to align to.
+	(setq column (+ column
+			;; Round up to a whole number of columns.
+			(* colwidth (ceiling length colwidth))))))))
+
+(defun completion--insert-vertical (strings group-fun
+                                            _length _wwidth
+                                            colwidth columns)
+  (let ((column 0)
+        (rows (/ (length strings) columns))
+	(row 0)
+	(last-title nil)
+	(last-string nil)
+        (start-point (point))
+        (next 0) (pos 0))
+    (dolist (str strings)
+      (unless (equal last-string str) ; Remove (consecutive) duplicates.
+	(setq last-string str)
+	(when (> row rows)
+	  (goto-char start-point)
+	  (setq row 0 column (+ column colwidth)))
+        (when group-fun
+          (let ((title (funcall group-fun (if (consp str) (car str) str) nil)))
+            (unless (equal title last-title)
+              (setq last-title title)
+              (when title
+                ;; Align before title insertion
+	        (when (> column 0)
+	          (end-of-line)
+	          (while (> (current-column) column)
+	            (if (eobp)
+		        (insert "\n")
+	              (forward-line 1)
+	              (end-of-line)))
+	          (insert " \t")
+	          (set-text-properties (1- (point)) (point)
+			               `(display (space :align-to ,column))))
+                (let* ((fmt completions-group-format)
+                       (len (length fmt)))
+                  ;; Adjust display space for columns
+                  (when (equal (get-text-property (- len 1) 'display fmt) '(space :align-to right))
+                    (setq fmt (substring fmt))
+                    (put-text-property (- len 1) len
+                                       'display
+                                       `(space :align-to ,(+ colwidth column -1))
+                                       fmt))
+                  (insert (format fmt title)))
+                ;; Align after title insertion
+	        (if (> column 0)
+	            (forward-line)
+	          (insert "\n"))))))
+        ;; Align before candidate insertion
+	(when (> column 0)
+	  (end-of-line)
+	  (while (> (current-column) column)
+	    (if (eobp)
+		(insert "\n")
+	      (forward-line 1)
+	      (end-of-line)))
+	  (insert " \t")
+	  (set-text-properties (1- (point)) (point)
+			       `(display (space :align-to ,column))))
+        (completion--insert str group-fun)
+        ;; Align after candidate insertion
+	(if (> column 0)
+	    (forward-line)
+	  (insert "\n"))
+	(setq row (1+ row))))))
+
+(defun completion--insert-one-column (strings group-fun &rest _)
+  (let ((last-title nil) (last-string nil))
+    (dolist (str strings)
+      (unless (equal last-string str) ; Remove (consecutive) duplicates.
+	(setq last-string str)
+        (when group-fun
+          (let ((title (funcall group-fun (if (consp str) (car str) str) nil)))
+            (unless (equal title last-title)
+              (setq last-title title)
+              (when title
+                (insert (format completions-group-format title) "\n")))))
+        (completion--insert str group-fun)
+        (insert "\n")))))
+
+(defun completion--insert (str group-fun)
+  (if (not (consp str))
+      (add-text-properties
+       (point)
+       (progn
+         (insert
+          (if group-fun
+              (funcall group-fun str 'transform)
+            str))
+         (point))
+       `(mouse-face highlight completion--string ,str))
+    ;; If `str' is a list that has 2 elements,
+    ;; then the second element is a suffix annotation.
+    ;; If `str' has 3 elements, then the second element
+    ;; is a prefix, and the third element is a suffix.
+    (let* ((prefix (when (nth 2 str) (nth 1 str)))
+           (suffix (or (nth 2 str) (nth 1 str))))
+      (when prefix
+        (let ((beg (point))
+              (end (progn (insert prefix) (point))))
+          (put-text-property beg end 'mouse-face nil)))
+      (completion--insert (car str) group-fun)
+      (let ((beg (point))
+            (end (progn (insert suffix) (point))))
+        (put-text-property beg end 'mouse-face nil)
+        ;; Put the predefined face only when suffix
+        ;; is added via annotation-function without prefix,
+        ;; and when the caller doesn't use own face.
+        (unless (or prefix (text-property-not-all
+                            0 (length suffix) 'face nil suffix))
+          (font-lock-prepend-text-property
+           beg end 'face 'completions-annotations))))))
 
 (defvar completion-setup-hook nil
   "Normal hook run at the end of setting up a completion list buffer.
-- 
2.20.1


[-- Attachment #4: 0003-minibuffer-completion-help-Do-not-check-completions-.patch --]
[-- Type: text/x-diff, Size: 1379 bytes --]

From 891ee2b7a50f69d75dd0a5624483457fc2c8e003 Mon Sep 17 00:00:00 2001
From: Daniel Mendler <mail@daniel-mendler.de>
Date: Sun, 2 May 2021 15:50:08 +0200
Subject: [PATCH 3/5] (minibuffer-completion-help): Do not check
 `completions-group` centrally

The guard variable `completions-group` should be checked in each
completion table individually.  The guard variable
`completions-detailed` variable is used in the same way.

* minibuffer.el (minibuffer-completion-help): Remove check of
  `completions-group`.
---
 lisp/minibuffer.el | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/lisp/minibuffer.el b/lisp/minibuffer.el
index 8b3f332782..bbf60899e2 100644
--- a/lisp/minibuffer.el
+++ b/lisp/minibuffer.el
@@ -2182,8 +2182,7 @@ minibuffer-completion-help
                           (plist-get completion-extra-properties
                                      :affixation-function)))
              (sort-fun (completion-metadata-get all-md 'display-sort-function))
-             (group-fun (and completions-group
-                             (completion-metadata-get all-md 'group-function)))
+             (group-fun (completion-metadata-get all-md 'group-function))
              (mainbuf (current-buffer))
              ;; If the *Completions* buffer is shown in a new
              ;; window, mark it as softly-dedicated, so bury-buffer in
-- 
2.20.1


[-- Attachment #5: 0004-completion-insert-vertical-Separate-groups-completel.patch --]
[-- Type: text/x-diff, Size: 5050 bytes --]

From 6a15c420c35c2657c3c53a862e891c21d9972d4f Mon Sep 17 00:00:00 2001
From: Daniel Mendler <mail@daniel-mendler.de>
Date: Sun, 2 May 2021 16:19:42 +0200
Subject: [PATCH 4/5] (completion--insert-vertical): Separate groups completely

Insert the candidates vertically within the groups, but keep the
groups separate using the full width group separators.

* minibuffer.el (completion--insert-vertical): Adjust grouping.
---
 lisp/minibuffer.el | 108 ++++++++++++++++++++-------------------------
 1 file changed, 48 insertions(+), 60 deletions(-)

diff --git a/lisp/minibuffer.el b/lisp/minibuffer.el
index bbf60899e2..2a2552a6f0 100644
--- a/lisp/minibuffer.el
+++ b/lisp/minibuffer.el
@@ -1869,66 +1869,54 @@ completion--insert-horizontal
 (defun completion--insert-vertical (strings group-fun
                                             _length _wwidth
                                             colwidth columns)
-  (let ((column 0)
-        (rows (/ (length strings) columns))
-	(row 0)
-	(last-title nil)
-	(last-string nil)
-        (start-point (point))
-        (next 0) (pos 0))
-    (dolist (str strings)
-      (unless (equal last-string str) ; Remove (consecutive) duplicates.
-	(setq last-string str)
-	(when (> row rows)
-	  (goto-char start-point)
-	  (setq row 0 column (+ column colwidth)))
-        (when group-fun
-          (let ((title (funcall group-fun (if (consp str) (car str) str) nil)))
-            (unless (equal title last-title)
-              (setq last-title title)
-              (when title
-                ;; Align before title insertion
-	        (when (> column 0)
-	          (end-of-line)
-	          (while (> (current-column) column)
-	            (if (eobp)
-		        (insert "\n")
-	              (forward-line 1)
-	              (end-of-line)))
-	          (insert " \t")
-	          (set-text-properties (1- (point)) (point)
-			               `(display (space :align-to ,column))))
-                (let* ((fmt completions-group-format)
-                       (len (length fmt)))
-                  ;; Adjust display space for columns
-                  (when (equal (get-text-property (- len 1) 'display fmt) '(space :align-to right))
-                    (setq fmt (substring fmt))
-                    (put-text-property (- len 1) len
-                                       'display
-                                       `(space :align-to ,(+ colwidth column -1))
-                                       fmt))
-                  (insert (format fmt title)))
-                ;; Align after title insertion
-	        (if (> column 0)
-	            (forward-line)
-	          (insert "\n"))))))
-        ;; Align before candidate insertion
-	(when (> column 0)
-	  (end-of-line)
-	  (while (> (current-column) column)
-	    (if (eobp)
-		(insert "\n")
-	      (forward-line 1)
-	      (end-of-line)))
-	  (insert " \t")
-	  (set-text-properties (1- (point)) (point)
-			       `(display (space :align-to ,column))))
-        (completion--insert str group-fun)
-        ;; Align after candidate insertion
-	(if (> column 0)
-	    (forward-line)
-	  (insert "\n"))
-	(setq row (1+ row))))))
+  (while strings
+    (let ((group nil)
+          (column 0)
+	  (row 0)
+          (rows)
+          (last-string nil))
+      (if group-fun
+          (let* ((str (car strings))
+                 (title (funcall group-fun (if (consp str) (car str) str) nil)))
+            (while (and strings
+                        (equal title (funcall group-fun
+                                              (if (consp (car strings))
+                                                  (car (car strings))
+                                                (car strings))
+                                              nil)))
+              (push (car strings) group)
+              (pop strings))
+            (setq group (nreverse group)))
+        (setq group strings
+              strings nil))
+      (setq rows (/ (length group) columns))
+      (when group-fun
+        (let* ((str (car group))
+               (title (funcall group-fun (if (consp str) (car str) str) nil)))
+          (when title
+            (goto-char (point-max))
+            (insert (format completions-group-format title) "\n"))))
+      (dolist (str group)
+        (unless (equal last-string str) ; Remove (consecutive) duplicates.
+	  (setq last-string str)
+	  (when (> row rows)
+            (forward-line (- -1 rows))
+	    (setq row 0 column (+ column colwidth)))
+	  (when (> column 0)
+	    (end-of-line)
+	    (while (> (current-column) column)
+	      (if (eobp)
+		  (insert "\n")
+	        (forward-line 1)
+	        (end-of-line)))
+	    (insert " \t")
+	    (set-text-properties (1- (point)) (point)
+			         `(display (space :align-to ,column))))
+          (completion--insert str group-fun)
+	  (if (> column 0)
+	      (forward-line)
+	    (insert "\n"))
+	  (setq row (1+ row)))))))
 
 (defun completion--insert-one-column (strings group-fun &rest _)
   (let ((last-title nil) (last-string nil))
-- 
2.20.1


[-- Attachment #6: 0005-minibuffer-completion-help-Add-group-sorting.patch --]
[-- Type: text/x-diff, Size: 3967 bytes --]

From 4601a270af996193c6ff06a3d32a035657bf5141 Mon Sep 17 00:00:00 2001
From: Daniel Mendler <mail@daniel-mendler.de>
Date: Tue, 11 May 2021 09:08:05 +0200
Subject: [PATCH 5/5] (minibuffer-completion-help): Add group sorting

Sort the groups as returned by the `group-function` of the completion
table depending on the value of the customizable variable
`completions-group-sort`.  By default `completions-group-sort` is set
to nil.  The variable can be set to the symbol `alphabetical` in order
to configure alphabetical sorting.  Furthermore, a custom sorting
function can be used as value of `completions-group-sort`.

* lisp/minibuffer.el (completions-group-sort): New variable.
(minibuffer--group-by): Add SORT-FUN argument.
(minibuffer-completion-help): Pass `completions-group-sort` to
`minibuffer--group-by`.
---
 lisp/minibuffer.el | 39 +++++++++++++++++++++++++++++++++------
 1 file changed, 33 insertions(+), 6 deletions(-)

diff --git a/lisp/minibuffer.el b/lisp/minibuffer.el
index 2a2552a6f0..6bd3baf074 100644
--- a/lisp/minibuffer.el
+++ b/lisp/minibuffer.el
@@ -1147,10 +1147,22 @@ completion-cycle-threshold
 
 (defcustom completions-group nil
   "Enable grouping of completion candidates in the *Completions* buffer.
-See also `completions-group-format'."
+See also `completions-group-format' and `completions-group-sort'."
   :type 'boolean
   :version "28.1")
 
+(defcustom completions-group-sort nil
+  "Sort groups in the *Completions* buffer.
+
+The value can either be nil to disable sorting, `alphabetical' for
+alphabetical sorting or a custom sorting function.  The sorting
+function takes and returns an alist of groups, where each element is a
+pair of a group title string and a list of group candidate strings."
+  :type '(choice (const nil :tag "No sorting")
+                 (const alphabetical :tag "Alphabetical sorting")
+                 function)
+  :version "28.1")
+
 (defcustom completions-group-format
   (concat
    (propertize "    " 'face 'completions-group-separator)
@@ -1434,16 +1446,21 @@ minibuffer--sort-preprocess-history
                      (substring c base-size)))
                  hist)))))
 
-(defun minibuffer--group-by (fun elems)
-  "Group ELEMS by FUN."
+(defun minibuffer--group-by (group-fun sort-fun elems)
+  "Group ELEMS by GROUP-FUN and sort groups by SORT-FUN."
   (let ((groups))
     (dolist (cand elems)
-      (let* ((key (funcall fun cand nil))
+      (let* ((key (funcall group-fun cand nil))
              (group (assoc key groups)))
         (if group
             (setcdr group (cons cand (cdr group)))
           (push (list key cand) groups))))
-    (mapcan (lambda (x) (nreverse (cdr x))) (nreverse groups))))
+    (setq groups (nreverse groups)
+          groups (mapc (lambda (x)
+                         (setcdr x (nreverse (cdr x))))
+                       groups)
+          groups (funcall sort-fun groups))
+    (mapcan #'cdr groups)))
 
 (defun completion-all-sorted-completions (&optional start end)
   (or completion-all-sorted-completions
@@ -2216,7 +2233,17 @@ minibuffer-completion-help
                       ;; `group-function'.
                       (when group-fun
                         (setq completions
-                              (minibuffer--group-by group-fun completions)))
+                              (minibuffer--group-by
+                               group-fun
+                               (pcase completions-group-sort
+                                 ('nil #'identity)
+                                 ('alphabetical
+                                  (lambda (groups)
+                                    (sort groups
+                                          (lambda (x y)
+                                            (string< (car x) (car y))))))
+                                 (_ completions-group-sort))
+                               completions)))
 
                       (cond
                        (aff-fun
-- 
2.20.1