Completion: display of candidates

Completion: display of candidates

[Tassilo Horn]

Hi all,

I have a function of one argument which computes a list of completions
matching the given argument.  I can then use

  (completing-read "Prompt: " (completion-table-dynamic #'my-function))

in order to read with completion from the minibuffer.

So far, so good, but I have two questions:

1. The docs for completion-table-dynamic say that the given function
   should return an alist (not just a list) but doesn't describe the
   structure of the entries.  Neither do I find a description in the
   info docs.  So what should it be?

2. Is there a way to display and complete on a different representation
   than the actual completion candidates?  In my case, my-func returns
   absolute file names but I'd like to complete on just the basename,
   and display the entries in the form of "basename (directory)".

Bye,
Tassilo

Re: Completion: display of candidates

[Stefan Monnier]

> 1. The docs for completion-table-dynamic say that the given function
>    should return an alist

That's an error: the return value should be a completion table.

> 2. Is there a way to display and complete on a different representation
>    than the actual completion candidates?  In my case, my-func returns
>    absolute file names but I'd like to complete on just the basename,
>    and display the entries in the form of "basename (directory)".

You can modify `my-func` to return file names in the form
"basename (directory)", no?


        Stefan

Re: Completion: display of candidates

[Tassilo Horn]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> 1. The docs for completion-table-dynamic say that the given function
>>    should return an alist
>
> That's an error: the return value should be a completion table.

So a "list of strings or cons cells, an obarray, a hash table, or a
completion function".  When you know what to look for, you'll find it
(in the info docs for try-completion).

>> 2. Is there a way to display and complete on a different representation
>>    than the actual completion candidates?  In my case, my-func returns
>>    absolute file names but I'd like to complete on just the basename,
>>    and display the entries in the form of "basename (directory)".
>
> You can modify `my-func` to return file names in the form
> "basename (directory)", no?

Yes, I could.  I just thought that there's probably some kind of
metadata I could attach for customizing the display.

So I guess the best thing to do is to format the entries as you suggest
and put a text property on the text holding the absolute path so that I
don't have to reconstruct it myself which could be error-prone if the
directory or basename contain parens themselves.

Bye,
Tassilo

Re: Completion: display of candidates

[Stefan Monnier]

>>> 1. The docs for completion-table-dynamic say that the given function
>>>    should return an alist
>> That's an error: the return value should be a completion table.
> So a "list of strings or cons cells, an obarray, a hash table, or a
> completion function".

Right.  With the caveat that this function will not be consulted for
`completion-metadata` and `completion-boundaries` queries.

I just pushed a corresponding patch to `emacs-26`.

> When you know what to look for, you'll find it
> (in the info docs for try-completion).

It should also be described in `(elisp)Programmed Completion', as
suggested in the docstring.

> Yes, I could.  I just thought that there's probably some kind of
> metadata I could attach for customizing the display.

It can't be "display only" since the user expects to type what is
displayed, so it also affects what the user types.

BTW, you might be interested in the GNU ELPA package `uniquify-files`
which tries to do almost exactly what you suggest (it requires a few
ugly hacks currently, but we hope to be able to improve the
minibuffer.el infrastructure so those can be dispensed with).

> So I guess the best thing to do is to format the entries as you suggest
> and put a text property on the text holding the absolute path so that I
> don't have to reconstruct it myself which could be error-prone if the
> directory or basename contain parens themselves.

You can't really rely on those text-properties, because the user may
have typed that text by hand.


        Stefan

Re: Completion: display of candidates

[Tassilo Horn]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>> Yes, I could.  I just thought that there's probably some kind of
>> metadata I could attach for customizing the display.
>
> It can't be "display only" since the user expects to type what is
> displayed, so it also affects what the user types.

Right, and here comes the next problem: concretely I get my completions
from `locate --basename <pattern>`.  So the user (me) might enter a
wildcard pattern like "foo*bar.*".  But the completions/matches
obviously have no * in it, so no completion matches the candidates.

How to handle that?  Use the PREDICATE argument in completing-read so
that the input ".em*s" matches the found candidate ".emacs (/home/horn)"
by transforming the wildcard to an emacs regexp and using
string-match-p?

> BTW, you might be interested in the GNU ELPA package `uniquify-files`
> which tries to do almost exactly what you suggest (it requires a few
> ugly hacks currently, but we hope to be able to improve the
> minibuffer.el infrastructure so those can be dispensed with).

I'll have a look at it.  Thanks for the pointer.

>> So I guess the best thing to do is to format the entries as you
>> suggest and put a text property on the text holding the absolute path
>> so that I don't have to reconstruct it myself which could be
>> error-prone if the directory or basename contain parens themselves.
>
> You can't really rely on those text-properties, because the user may
> have typed that text by hand.

I use the REQUIRE-MATCH argument of completing read, and even if I
didn't, if there's no property the user (me) must have entered a file
name then.

Bye,
Tassilo

Re: Completion: display of candidates

[Stefan Monnier]

> Right, and here comes the next problem: concretely I get my completions
> from `locate --basename <pattern>`.  So the user (me) might enter a
> wildcard pattern like "foo*bar.*".  But the completions/matches
> obviously have no * in it, so no completion matches the candidates.

That completely depends on the completion style.

For example `partial-completion` (which is included in the default
`completion-styles`) does accept * so you can do `M-x r*v*uf TAB` to
find revert-buffer.

> How to handle that?  Use the PREDICATE argument in completing-read so

The PREDICATE argument can only rule out matches, not add new ones.


        Stefan

Re: Completion: display of candidates

[Tassilo Horn]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>> Right, and here comes the next problem: concretely I get my
>> completions from `locate --basename <pattern>`.  So the user (me)
>> might enter a wildcard pattern like "foo*bar.*".  But the
>> completions/matches obviously have no * in it, so no completion
>> matches the candidates.
>
> That completely depends on the completion style.
>
> For example `partial-completion` (which is included in the default
> `completion-styles`) does accept * so you can do `M-x r*v*uf TAB` to
> find revert-buffer.

Hm, that there can be dependencies between (1) finding completion
candidates and (2) completion styles doesn't spark joy in my heart.  I
thought of (1) as a kind of generic backend and (2) as a frontend which
users select based on personal preference.  But since both have to work
with the user's input string, I don't see how to make it better...

>> How to handle that?  Use the PREDICATE argument in completing-read so
>
> The PREDICATE argument can only rule out matches, not add new ones.

Yeah, in the end I've waived my hands and went without text properties
and just selected a unicode character which is unlikely to be used in
file names as separator.  The results are quite satisfying.

--8<---------------cut here---------------start------------->8---
(defconst th/recentf-locate-excluded-paths
  (let ((home (getenv "HOME")))
    (list
     #'backup-file-name-p
     (expand-file-name ".cargo/" home)
     (expand-file-name ".cache/" home)
     (expand-file-name ".m2/" home)
     (expand-file-name ".IntelliJIdea[^/]+/" home))))

(defun th/recentf-locate-completions (str)
  (with-current-buffer (get-buffer-create " *th/locate-matches*")
    (erase-buffer)
    (mapc (lambda (rf) (insert rf "\n")) recentf-list)
    (let ((home-dir (getenv "HOME"))
	  lst line-move-visual)
      (when (> (length str) 2)
	(call-process "locate" nil t nil
		      "--basename"
		      "--existing"
		      "--ignore-case"
		      "--limit" "500"
		      str))
      (goto-char (point-min))
      (while (not (eobp))
	(let* ((path (buffer-substring (point) (line-end-position)))
	       (basename (file-name-nondirectory path))
	       (dir (file-name-directory path)))
	  (unless (seq-find (lambda (pred)
			      (cond
			       ((stringp pred)
				(string-match-p pred path))
			       ((functionp pred) (funcall pred path))
			       (t (error "Don't know how to handle %S" pred))))
			    th/recentf-locate-excluded-paths)
	    (push (format "%s ‼ %s" basename dir) lst)))
	(next-line))
      (sort lst (lambda (a b)
		  (or
		   ;; a is in HOME but b is not, so sort a before b
		   (and (string-match-p (concat " ‼ " home-dir) a)
			(not (string-match-p (concat " ‼ " home-dir) b)))
		   ;; otherwise sort by base name.
		   (string-lessp a b)))))))

(defun th/recentf-locate-file (locate-candidate)
  (interactive
   (list (completing-read
	  "Locate File: "
	  (completion-table-dynamic #'th/recentf-locate-completions)
	  nil t)))
  (let ((path (progn
		(string-match "^\\(.*\\) ‼ \\(.*\\)$" locate-candidate)
		(expand-file-name (match-string 1 locate-candidate)
				  (match-string 2 locate-candidate)))))
    (find-file path)))

(global-set-key (kbd "<f5>") #'th/recentf-locate-file)
--8<---------------cut here---------------end--------------->8---

Bye,
Tassilo

Re: Completion: display of candidates

[Stefan Monnier]

>> For example `partial-completion` (which is included in the default
>> `completion-styles`) does accept * so you can do `M-x r*v*uf TAB` to
>> find revert-buffer.
> Hm, that there can be dependencies between (1) finding completion
> candidates and (2) completion styles doesn't spark joy in my heart.

The completion table is not necessarily looked up with the user's typed
string.  It's meant to be nothing more than a standardized
representation of a *set* of candidates.  How this set is filtered on
the basis of the user's string is under the control of the
completion styles.

> I thought of (1) as a kind of generic backend and (2) as a frontend
> which users select based on personal preference.  But since both have
> to work with the user's input string, I don't see how to make it
> better...

The completion-table does not see "the user's input string".  It only
sees "a string" (chosen by the completion-style) and should return the
set of candidates which have that string as a prefix.  In some cases,
the string it will see is "" and it should hence return the complete
list of candidates (after which the completion style will perform its
own filtering on it).

> Yeah, in the end I've waived my hands and went without text properties
> and just selected a unicode character which is unlikely to be used in
> file names as separator.  The results are quite satisfying.

Great,


        Stefan

Re: Completion: display of candidates

[Stefan Monnier]

>> The completion-table does not see "the user's input string".  It only
>> sees "a string" (chosen by the completion-style) and should return the
>> set of candidates which have that string as a prefix.
> Is it always a prefix, e.g., also with the substring or flex completion
> style?

Yes.  Completion styles like substring/flex/partial-completion make do
with this as the underlying primitive.

>> In some cases, the string it will see is "" and it should hence return
>> the complete list of candidates (after which the completion style will
>> perform its own filtering on it).
> What's the best thing to do if the number of candidates is too large to
> deliver them all?

Good question.  The answer is somewhere around "it depends".

> Right now, my function passed to completion-table-dynamic always returns
> the elements of the recentf-list + the results of a locate --basename
> search, however, the latter is only triggered if "the string" is longer
> than 2 characters.  Thus, the completion-table grows when going from
> search string "fo" to "foo" which is probably not expected by completion
> styles.

Yes, it's a problem which will make (typically) `substring` and `flex`
misbehave: they always call the completion-table with "" and then do
their own filtering so they'll only see the recentf-list elements ;-)

That's one of the fundamental problems we should fix in the design of
the completion-table API.


        Stefan