completing-read return meta-information?

completing-read return meta-information?

[Stephen Leake]

I'd like completing-read to be able to return meta-information with the
completion string.

For example, consider completing a file name when there are duplicates:

dir1/
    file1.el
    file2.el

dir2/
    file1.el
    file2.el

The completion function is reading the disk; it knows the absolute path
of the file that the user chooses. However, that information is lost when
completing-read returns just "file1.el<dir1>" (a uniqified file name).

So the caller has to repeat some of the work of the completion function
to use the returned result.

I tried storing the directory info in a text property of the completion
string; that was not returned.

A similar situation occurs when completing in an alist; the completion
function uses assoc to read the alist, but after completing-read
returns, the caller must call assoc again.

`completion-metadata' doesn't seem designed for arbitrary user
information. And I'd rather not have to call the completion table
function again; that would typically require re-computing the
meta-information, or caching it somewhere.

Is there a way to return meta-information?

What would have to be changed in completing-read to make this possible?

-- 
-- Stephe

RE: completing-read return meta-information?

[Drew Adams]

> I tried storing the directory info in a text property of the
> completion string; that was not returned.

FWIW, I proposed years ago that a completion candidate (string) be
able to be returned propertized.  "Be able" means that this can be
controlled, e.g., by a variable.

Here is a thread about this from 2008, for instance:
http://lists.gnu.org/archive/html/emacs-devel/2008-06/msg01503.html
There were other threads too.

FWIW, in Icicles I've done this for nearly a decade.  I use this
feature to recuperate information associated with an individual
candidate.  A candidate can be as rich as you like and return as
much info as you like.

The changes to the Emacs code needed to allow this are simple.

> A similar situation occurs when completing in an alist; the completion
> function uses assoc to read the alist, but after completing-read
> returns, the caller must call assoc again.

See above - same thing.  Just put the full alist element on the
string as a property.  You can use the exact alist element:
`eq'-testable.  No loss of information.

> `completion-metadata' doesn't seem designed for arbitrary user
> information. And I'd rather not have to call the completion table
> function again; that would typically require re-computing the
> meta-information, or caching it somewhere.
> 
> Is there a way to return meta-information?
> What would have to be changed in completing-read to make this possible?

Simply return the propertized candidate as is.  Makes completion
far more useful.

Re: completing-read return meta-information?

[Stephen Leake]

Drew Adams <drew.adams@oracle.com> writes:

>> I tried storing the directory info in a text property of the
>> completion string; that was not returned.
>
> FWIW, I proposed years ago that a completion candidate (string) be
> able to be returned propertized.  "Be able" means that this can be
> controlled, e.g., by a variable.
>
> Here is a thread about this from 2008, for instance:
> http://lists.gnu.org/archive/html/emacs-devel/2008-06/msg01503.html
> There were other threads too.
>
> FWIW, in Icicles I've done this for nearly a decade.  I use this
> feature to recuperate information associated with an individual
> candidate.  A candidate can be as rich as you like and return as
> much info as you like.
>
> The changes to the Emacs code needed to allow this are simple.

Excellent. Can you post a patch?

Or you can add it to the feature branch I'll push soon;
scratch/project-find-file.

Another choice would be for completing-read to return a cons when the
completion-table returns an alist. That would also require a flag to
allow the new behavior, to avoid breaking current alist completion
tables. 

-- 
-- Stephe

RE: completing-read return meta-information?

[Drew Adams]

> >> I tried storing the directory info in a text property of the
> >> completion string; that was not returned.
> >
> > FWIW, I proposed years ago that a completion candidate (string) be
> > able to be returned propertized.  "Be able" means that this can be
> > controlled, e.g., by a variable.
> >
> > Here is a thread about this from 2008, for instance:
> > http://lists.gnu.org/archive/html/emacs-devel/2008-06/msg01503.html
> > There were other threads too.
> >
> > FWIW, in Icicles I've done this for nearly a decade.  I use this
> > feature to recuperate information associated with an individual
> > candidate.  A candidate can be as rich as you like and return as
> > much info as you like.
> >
> > The changes to the Emacs code needed to allow this are simple.
> 
> Excellent. Can you post a patch?

See the old threads.  Stefan knows full well how to do this.
It's not wanted.

The Icicles code that does it is here, if you're interested:
http://www.emacswiki.org/emacs/download/icicles-fn.el.  In Icicle
mode, `icicle-completing-read' substitutes for `completing-read'.
In particular, it doesn't remove text properties from the return
value.

---

It also binds `minibuffer-allow-text-properties' to `t'.  But
you will note from `C-h v' for that variable that it has no
effect in vanilla Emacs on text read using completion.  IOW,
you might hope that this would be sufficient to give `toto' a
propertized string value, but you would be wrong:

(defun foo ()
  (interactive)
  (let ((minibuffer-allow-text-properties  t)
	(aa (propertize "aa" 'face 'highlight))
	(bb (propertize "bb" 'face 'error))
	(cc (propertize "cc" 'face 'escape-glyph)))
    (completing-read "foo: " (list aa bb cc))))

(setq toto (foo))

Re: completing-read return meta-information?

[Stephen Leake]

Drew Adams <drew.adams@oracle.com> writes:

>> I tried storing the directory info in a text property of the
>> completion string; that was not returned.
>
> Here is a thread about this from 2008, for instance:
> http://lists.gnu.org/archive/html/emacs-devel/2008-06/msg01503.html

That thread includes a request from Stefan for a clear use case for
this. So let me make my case clear.

I'm implementing file name completion in projects. The details vary with
each backend. Two backends I've implemented so far look like this:

(cl-defgeneric project-find-file (prj filename)
  "Find FILENAME with completion in current project PRJ."
  (let* ((flat-path (project-flat-search-path prj))
	 (regexp (project-ignore-files-regexp prj))
	 (predicate
	  (lambda (filename)
            ;; FIXME: should call project-ignores here with each directory
	    (not (string-match regexp filename)))))

    ;; (project-ignore-files-regexp prj) matches filenames, not
    ;; uniquified filenames. So it must be applied in
    ;; `find-file-path-completion-table', not `completing-read'.
    (setq filename
	  (completing-read
	   "file: " ;; prompt
	   (completion-table-dynamic (apply-partially
              'find-file-path-completion-table
              flat-path predicate))
	   nil
	   t ;; require match
	   filename
	   ))

    ;; We construct a relative path to ensure the filename is found on
    ;; `flat-path'.
    (when (string-match find-file-uniquify-regexp filename)
	(let ((dir (match-string 2 filename))
	      (prefix "../")
	      (i 0))

	  (while (< i (length dir))
	    (when (= (aref dir i) ?/)
	      (setq prefix (concat prefix "../")))
	    (setq i (1+ i)))

	  (setq filename
		(concat prefix
			dir
			"/"
			(match-string 1 filename)))
	  ))

    (let ((absfilename (locate-file filename flat-path nil)))
      (if absfilename
	  (find-file absfilename)
	;; FIXME: need human-readable name for project
	(error "'%s' not found in project." filename)))
    ))


(defun find-file-complete-global (filename)
  "Prompt for completion of FILENAME in a Gnu global project."
    (setq filename
	  (completing-read
	   "file: " ;; prompt
	   (completion-table-with-cache #'find-file-complete-global-table) ;; collection
	   nil ;; predicate
	   t ;; require match
	   filename
	   ))

    (when (string-match find-file-uniquify-regexp filename)
      ;; Get partial dir from conflict
      (setq filename (concat (match-string 2 filename) (match-string 1 filename))))

    ;; If there are two files like:
    ;;
    ;; src/keyboard.c
    ;; test/etags/c-src/emacs/src/keyboard.c
    ;;
    ;; and the user completes to the first, the following global call
    ;; will return both. The desired result is always the shortest.
    (with-current-buffer (cedet-gnu-global-call (list "--ignore-case" "-Pa" filename))
      (let ((paths (split-string (buffer-substring (point-min) (point-max)) "\n" t)))
	(setq paths (sort paths (lambda (a b) (< (length a) (length b)))))
	(car paths)))

    )


There is a desire to refactor this so that the only difference is inside
the completion table.

Currently, that is not possible, because completing-read does not return
the absolute path of the file that the user selected. Therefore the
calling code must do additional work, that is different for the two
backends. That work typically duplicates some of what the completion
table has already done.

The solution is to make completing-read return more information.

One way to do this with the current completing read code is to add the
entire path to the completion string, as a suffix. But that results in a
horrible user experience; in the Emacs project on my disk, "locate"
would complete to:

locate{.rnc</Projects/emacs/master/etc/schema/>
.el</Projects/emacs/master/lisp/cedet/ede/>
.elc</Projects/emacs/master/lisp/cedet/ede/>
.el</Projects/emacs/master/lisp/>
.elc</Projects/emacs/master/lisp/>}

instead of the mininmal:

locate{.rnc .el<lisp/cedet/ede/> .elc<lisp/cedet/ede/> .el<lisp/> .elc</lisp/>}


So a better solution is to allow completing-read to return the
additional information in some other way.

Three ways seem straightforward:

1) The completion table can add the absolute path in an `abspath' text
   property of the completion string.

    The rest of the completion functions must preserve the text
    property.

    This may need a new optional arg to completing-read, if current code
    relies on text properties being discarded.

2) The completion table can return an alist (note this is already
   supported), with the cdr of each element being the absolute path. In
   this case, completing-read (and other completion functions?) returns
   the selected cons.

    To avoid breaking current alist completion-tables, this needs a new
    optional argument to completing-read (and other completion
    functions); cons-result, nil by default.

    Currently, when a completion-table returns an alist, the predicate
    supplied to completing-read is applied to the cdr of the elements.
    That could be useful for the file completion case as well, although
    it will probably be more efficient to apply the predicate inside the
    table, to prune the file tree.
    
3) Extend completion-metadata to cover this usage; then the calling code
   calls completion-metadata with the completing-read string result to
   retrieve the absolute path.

    This has the downside that the completion table code must repeat
    work done previously to recompute the metadata.


The calling code then has a standard way of retrieving the absolute path
from the result of completing-read (either get the `abspath' text
property, take the cdr, or call completion-metadata), and can be
independent of the completion backend.

I think the alist approach is more in keeping with general Emacs style,
although xref-read-identifier is one recent precendent for using text
properties in a similar way.

Which is the best approach depends mostly on how much code inside the
completion functions must be changed.

-- 
-- Stephe

Re: completing-read return meta-information?

[Stefan Monnier]

> There is a desire to refactor this so that the only difference is inside
> the completion table.

I think the uniquification shouldn't be done in the completion table.

All the fancy rewriting to make things shorter should be done in the
completion-style, instead.  The style code can choose to return different
kinds of results for completion-all-completions (which gets displayed
in *Completions*, in Company, or in Icomplete-mode and where we should
try and reduce redundancy) and for completion-try-completion.

So completion-all-completions can return names like "foo<bar>" while
completion-try-completion can return "~/src/file/foo/toto/bar.agda".

Another option is to use a completely different completion code
(e.g. that's what filecache.el does).

> 1) The completion table can add the absolute path in an `abspath' text
>    property of the completion string.

I generally don't like the idea that the same text would fail to behave
in the same way when written by hand by the user (in which case it
won't have that text-property).


        Stefan

Re: completing-read return meta-information?

[Stephen Leake]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> There is a desire to refactor this so that the only difference is inside
>> the completion table.
>
> I think the uniquification shouldn't be done in the completion table.
>
> All the fancy rewriting to make things shorter should be done in the
> completion-style, instead.  The style code can choose to return different
> kinds of results for completion-all-completions (which gets displayed
> in *Completions*, in Company, or in Icomplete-mode and where we should
> try and reduce redundancy) and for completion-try-completion.
>
> So completion-all-completions can return names like "foo<bar>" while
> completion-try-completion can return "~/src/file/foo/toto/bar.agda".

And I gather completing-read returns the result of
completion-try-completion.

Ok, I'll see if I can figure out how to implement completion style code.

-- 
-- Stephe

Re: completing-read return meta-information?

[Stefan Monnier]

> And I gather completing-read returns the result of
> completion-try-completion.

Not really, no.  It returns the content of the buffer when the user hit
RET, mostly (unless the completion is of kind `require-match', in which
case that is indeed passed one last time to completion-try-completion).

The user could hit RET without having called any completion commands.


        Stefan

Re: completing-read return meta-information?

[Dmitry Gutov]

On 09/17/2015 04:24 PM, Stefan Monnier wrote:

> Not really, no.  It returns the content of the buffer when the user hit
> RET, mostly (unless the completion is of kind `require-match', in which

In find-file-in-project, it most likely will be. A non-existing or 
abbreviated file name input would be pretty useless for that command.

Re: completing-read return meta-information?

[Stefan Monnier]

>> Not really, no.  It returns the content of the buffer when the user hit
>> RET, mostly (unless the completion is of kind `require-match', in which
> In find-file-in-project, it most likely will be. A non-existing or
> abbreviated file name input would be pretty useless for that command.

IIRC, even for require-match we don't always call
completion-try-completion (we only do it if test-completion fails).


        Stefan

Re: completing-read return meta-information?

[Stephen Leake]

[-- Attachment #1: Type: text/plain, Size: 2647 bytes --]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> There is a desire to refactor this so that the only difference is inside
>> the completion table.
>
> I think the uniquification shouldn't be done in the completion table.
>
> All the fancy rewriting to make things shorter should be done in the
> completion-style, instead.  The style code can choose to return different
> kinds of results for completion-all-completions (which gets displayed
> in *Completions*, in Company, or in Icomplete-mode and where we should
> try and reduce redundancy) and for completion-try-completion.
>
> So completion-all-completions can return names like "foo<bar>" while
> completion-try-completion can return "~/src/file/foo/toto/bar.agda".

Attached is code that implements this, together with ert test code for
it (helpful in getting the details right).

This achieves the goal of moving all of the uniquification code out of
the completion table; that required advice on test-completion.

It also achieves the goal of returning an absolute string from
completing-read; that required advice on completing-read-default to call
the new function completion-get-data-string.

Note that this does not handle completion on multiple fields (for
uniquifying directories); I was focusing on the changes to completion
needed to return an absolute file name. That will be added later, and
may require design changes

See the header comment in minibuffer-patches.el for some design notes.

This may be an acceptable implementation, but it would be better to
extend completion-style-alist to include slots for
completion-test-completion and completion-get-data-string. I think that
can be done in a backward-compatible way; if (nth 2
completion-style-alist) (for completion-test-completion) returns a
string, just call test-completion, and if (nth 3 completion-style-alist)
(for completion-get-data-string) returns nil just return the input
string.

Then we can change all uses of `test-completion' to call
`completion-test-completion'.

I'm not entirely sure where the call to completion-get-data-string
should be added; one place is in `completion--complete-and-exit', but
there may be others. (I didn't advise completion--complete-and-exit
directly because I didn't want to mix advice and signals).


To try this out, load minibuffer-patches, and call find-absfile
via M-: (find-absfile load-path).

Note that will not allow you to complete to ede/locate.el; that will
also be added later.

When you enter a string that does not match any filename, this gives an
error from one of the basic completion functions; I have not tried to
track that down yet.

--
-- Stephe

[-- Attachment #2: minibuffer-patches.el --]
[-- Type: application/emacs-lisp, Size: 12896 bytes --]

[-- Attachment #3: minibuffer-test.el --]
[-- Type: application/emacs-lisp, Size: 6721 bytes --]

Re: completing-read return meta-information?

[Stephen Leake]

Stephen Leake <stephen_leake@stephe-leake.org> writes:

> To try this out, load minibuffer-patches, and call find-absfile
> via M-: (find-absfile load-path).
>
> Note that will not allow you to complete to ede/locate.el; that will
> also be added later.

To play with completion, use an extended path:

M-: (setq test-path (append load-path '("c:/Projects/emacs/master/lisp/cedet/ede"))

M-: (find-absfile load-path)

Now entering "locate" will show both lisp/locate.el and
lisp/cedet/ede/locate.el

It also shows the .elc files; I need to a filter predicate.

-- 
-- Stephe

Re: completing-read return meta-information?

[Stefan Monnier]

> It also achieves the goal of returning an absolute string from
> completing-read; that required advice on completing-read-default to call
> the new function completion-get-data-string.

Why can't you make completion-try-completion return an absolute filename
when there's only one match?  That would eliminate the need for
those advices.


        Stefan


PS: Also, as a user, I think I'd rather see names like "dir/file" than
"file<dir>".

Re: completing-read return meta-information?

[Stephen Leake]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>> It also achieves the goal of returning an absolute string from
>> completing-read; that required advice on completing-read-default to call
>> the new function completion-get-data-string.
>
> Why can't you make completion-try-completion return an absolute filename
> when there's only one match?  

The string returned from completion-try-completion is displayed in the
minibuffer, as the completion prefix. I don't want that to be the
display string.

Part of the point of using path completion is that you don't care what
directory the file is in. So having the absolute path displayed as the
final completion feels like the system forcing you to care.

More importantly, the user can type <ret> at any time; then
test-completion should return t when it is passed the abbreviated
display string from the first completion; that is known to be a unique
valid completion.

So this approach would require the user to always type tab, or go thru
one more confirmation step after typing <ret>. I tried this approach
early in this work, and could not make it work consistently.


This discussion points out that "user input string" and "display string"
have the same format; both can be passed to completion-try-completion and
completion-test-completion. The only difference is that the strings
returned by completion-all-completions all identifiy unique, valid
files. So I think I'll merge those two string types to "user string";
that should make this clearer.

> PS: Also, as a user, I think I'd rather see names like "dir/file" than
> "file<dir>".

You only need directory names when there are conflicting names; on many
paths, there will be none or few.

I mainly used that style because it meant the user string is a prefix of
all the completions, which some of the completion primitives expect, so
it made things simpler.

On the other hand, I use the postfix uniquify style for buffer names, so
I'm used to it :).

I'll try to implement directory completion in both styles; this should
be a user preference, just as it is in buffer names.

-- 
-- Stephe

Re: completing-read return meta-information?

[Stephen Leake]

[-- Attachment #1: Type: text/plain, Size: 1605 bytes --]

Stephen Leake <stephen_leake@stephe-leake.org> writes:

> Stefan Monnier <monnier@IRO.UMontreal.CA> writes:
>
>>> It also achieves the goal of returning an absolute string from
>>> completing-read; that required advice on completing-read-default to call
>>> the new function completion-get-data-string.
>>
>> Why can't you make completion-try-completion return an absolute filename
>> when there's only one match?  
>
> The string returned from completion-try-completion is displayed in the
> minibuffer, as the completion prefix. I don't want that to be the
> display string.
>
> Part of the point of using path completion is that you don't care what
> directory the file is in. So having the absolute path displayed as the
> final completion feels like the system forcing you to care.
>
> More importantly, the user can type <ret> at any time; then
> test-completion should return t when it is passed the abbreviated
> display string from the first completion; that is known to be a unique
> valid completion.
>
> So this approach would require the user to always type tab, or go thru
> one more confirmation step after typing <ret>. I tried this approach
> early in this work, and could not make it work consistently.

I took another stab at this, and got it to work; attached.

The version with advice is cleaner, but not by a whole lot.

The completion function calls a deuniquify function, and the top level
client code calls `try-completion' after calling `completing-read'. The
user sees an absolute filename in the completion buffer only if they
type <tab> twice on a unique file.

-- 
-- Stephe

[-- Attachment #2: minibuffer-patches-2.el --]
[-- Type: application/emacs-lisp, Size: 13728 bytes --]

Re: completing-read return meta-information?

[Stefan Monnier]

>>> Why can't you make completion-try-completion return an absolute filename
>>> when there's only one match?  
>> The string returned from completion-try-completion is displayed in the
>> minibuffer, as the completion prefix.

That's right (tho only once it's unique).

>> I don't want that to be the display string.

I think that's a mistake.

>> Part of the point of using path completion is that you don't care what
>> directory the file is in.  So having the absolute path displayed as the
>> final completion feels like the system forcing you to care.

No, you still don't have to type it, so you're not forced to care.
But yes, you get to see it, so you can (even sub-consciously)
double-check that it's indeed the one you want.

>> More importantly, the user can type <ret> at any time; then
>> test-completion should return t when it is passed the abbreviated
>> display string from the first completion; that is known to be a unique
>> valid completion.

If the call to completing-read uses `require-match' (which I'd expect to
be the case), then hitting RET at any time should work fine since
if the file name is incomplete test-completion will return nil, so we'll
call completion-try-completion.

[ And depending on the exact value of `require-match' the
  expanded/absolute filename will be shown to the user or not.  ]

>	   (cons 'path path))))

IIUC this variable `path' holds a file name, not a list of directories,
so the GNU Coding Standard would not want to use "path" here (since
this word is only used for a list of directories as in load-path).
You could use `filename' instead.


        Stefan

Re: completing-read return meta-information?

[Dmitry Gutov]

On 09/27/2015 06:45 PM, Stefan Monnier wrote:

> No, you still don't have to type it, so you're not forced to care.
> But yes, you get to see it, so you can (even sub-consciously)
> double-check that it's indeed the one you want.

I also think that it's a benefit, rather than a drawback.

Re: completing-read return meta-information?

[Stephen Leake]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>	   (cons 'path path))))
>
> IIUC this variable `path' holds a file name, not a list of
> directories,

No, it is a list of directories. ie load-path for elisp code (possibly
exteneded with lisp/cedet/**)

-- 
-- Stephe

Re: completing-read return meta-information?

[Stephen Leake]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>>> Why can't you make completion-try-completion return an absolute filename
>>>> when there's only one match?  
>>> The string returned from completion-try-completion is displayed in the
>>> minibuffer, as the completion prefix.
>
> That's right (tho only once it's unique).
>
>>> I don't want that to be the display string.
>
> I think that's a mistake.

The user can use `locate-file', if they care about the directories.

We could enhance that completion table function to deal with duplicate
filenames.

-- 
-- Stephe

Re: completing-read return meta-information?

[Stefan Monnier]

>>>> I don't want that to be the display string.
>> I think that's a mistake.
> The user can use `locate-file', if they care about the directories.

I still don't see why you're so intent on preventing the user from
seeing the real file name.

Yes, we want the user to find the file without having to know where it
is, but that's quite different from actively trying to hide
that location.


        Stefan

Re: completing-read return meta-information?

[Dmitry Gutov]

On 09/22/2015 06:21 PM, Stephen Leake wrote:

>> PS: Also, as a user, I think I'd rather see names like "dir/file" than
>> "file<dir>".
>
> You only need directory names when there are conflicting names; on many
> paths, there will be none or few.

I think the key distinction here is between "need to input" and "need to 
see". As long as the user doesn't need to input the directory, I think 
it's fine if they end up seeing it at some point.

> I mainly used that style because it meant the user string is a prefix of
> all the completions, which some of the completion primitives expect, so
> it made things simpler.

This sounds like it'll filter out all results if the user *does* input 
the directory, in all or at least some of the cases.

Please think back to the examples I gave in the other thread.

> On the other hand, I use the postfix uniquify style for buffer names, so
> I'm used to it :).
>
> I'll try to implement directory completion in both styles; this should
> be a user preference, just as it is in buffer names.

Having it customizable would be ideal, but I'm not sure it's possible to 
have uniquify-style completions without excluding some valid usage 
scenarios, like mentioned above.