how to determine the current table (really) being used for minibuffer completion?

how to determine the current table (really) being used for minibuffer completion?

[Drew Adams]

Since the Emacs 23 changes to minibuffer completion, there is little doc, and
there are few doc strings. It's not always obvious how to determine the current
state of evaluation.

In particular, when completion tables are tried in sequence under the covers,
how is it possible to know at some point which table is actually being tried?
For example, how to know, during file-name completion, whether the table
currently being tried is `(completion--make-envvar-table)' or
`(completion--file-name-table)'?

We have the global variable `minibuffer-completion-table', but that is
apparently useless in this context. That var might be bound to some function
`foo', but that doesn't mean that it is `foo' that is actually trying to perform
completion at the moment, since completion now can involve several completion
attempts using different tables (e.g. functions), successively.

For example, the table (function) `read-file-name-internal', which is the value
of `minibuffer-completion-table', is now an alias for the closure returned by
this:

(completion-table-in-turn 'completion--embedded-envvar-table
                          'completion--file-name-table)

So not only is checking `minibuffer-completion-table' against
`read-file-name-internal' useless, there is no way to know which of the two
tables (functions) tried in turn is actually being used at a given time. Unless
I'm missing something.

How about (at least) recording in some global var the table that is currently
being used/tried?

Re: how to determine the current table (really) being used for minibuffer completion?

[Stefan Monnier]

> Since the Emacs 23 changes to minibuffer completion, there is little
> doc, and there are few doc strings.  It's not always obvious how to
> determine the current state of evaluation.

I do not know what you mean.  Please give a concrete example showing how
you used to do it in Emacs-22.

> In particular, when completion tables are tried in sequence under the
> covers, how is it possible to know at some point which table is
> actually being tried?

If you mean uses of completion-table-in-turn: you can't and neither
could you in Emacs-22; and this is not strictly a change in the
minibuffer completion but in the implementation of some completion
tables (yes, I made changes to both because the minibuffer completion
changes needed some adjustments in the competion tables to work better,
and because I wanted to rewrite in Lisp the C implementation of some of
the completion tables, and because I wanted to give the code more
structure).

> For example, how to know, during file-name completion, whether the table
> currently being tried is `(completion--make-envvar-table)' or
> `(completion--file-name-table)'?

You can't (and neither could you before, AFAICT).
Could you give us some context to beter understand when you need it?

> We have the global variable `minibuffer-completion-table', but that is
> apparently useless in this context. That var might be bound to some
> function `foo', but that doesn't mean that it is `foo' that is
> actually trying to perform completion at the moment, since completion
> now can involve several completion attempts using different tables
> (e.g. functions), successively.

It does mean that `foo' is the table that does the completion.
That table may be decomposed into several distinct cases, but that's
nothing new.  The same was already the case when completing for
Info-goto-node, for example.

> So not only is checking `minibuffer-completion-table' against
> `read-file-name-internal' useless,

Comparing functions is usually a bad idea.  Sometimes, there's not much
else we can do, admittedly.

> How about (at least) recording in some global var the table that is
> currently being used/tried?

It's not even clear what that would mean.


        Stefan

RE: how to determine the current table (really) being used for minibuffer completion?

[Drew Adams]

> > Since the Emacs 23 changes to minibuffer completion, there is little
> > doc, and there are few doc strings.  It's not always obvious how to
> > determine the current state of evaluation.
> 
> I do not know what you mean.  Please give a concrete example 
> showing how you used to do it in Emacs-22.

How I used to do what? There is no concrete example.

The point is that it is impossible (difficult?) to know which kind of completion
function is currently being attempted, or which type completed successfully.
Previously, comparing `minibuffer-completion-table' against a given function
gave some idea.

And surely you know what I mean wrt doc and doc strings. ;-)

> > In particular, when completion tables are tried in sequence 
> > under the covers, how is it possible to know at some point
> > which table is actually being tried?
> 
> If you mean uses of completion-table-in-turn: you can't and neither
> could you in Emacs-22; and this is not strictly a change in the
> minibuffer completion but in the implementation of some completion
> tables (yes, I made changes to both because the minibuffer completion
> changes needed some adjustments in the competion tables to 
> work better, and because I wanted to rewrite in Lisp the C implementation 
> of some of the completion tables, and because I wanted to give the
> code more structure).

So you've confirmed what I said: one cannot know. The request is to be able to
know. IOW, to be able to have some finer-grain knowledge of the completion
state.

> > For example, how to know, during file-name completion, 
> > whether the table currently being tried is
> > `(completion--make-envvar-table)' or
> > `(completion--file-name-table)'?
> 
> You can't (and neither could you before, AFAICT).
> Could you give us some context to beter understand when you need it?

No. I didn't say I needed it in some concrete way.

The point is that now there can be radically different kinds of completion that
are used, all within the same `read-file-name-internal' (for example). I would
like to be able to know which phase of that completion processing I'm in, or
which phase succeeded if completion was successful.

I would like, for instance (just an example), to be able to distinguish
file-name completion per se from env var completion, both of which are now
included as part of "file-name" completion in a larger sense.

> > We have the global variable `minibuffer-completion-table', 
> > but that is apparently useless in this context. That var might
> > be bound to some function `foo', but that doesn't mean that
> > it is `foo' that is actually trying to perform completion
> > at the moment, since completion now can involve several
> > completion attempts using different tables
> > (e.g. functions), successively.
> 
> It does mean that `foo' is the table that does the completion.

Only in a general way. `foo' or `read-file-name-internal' is the overall
completion function, to be sure, but the completion processing is now decomposed
into several distinct cases (as you say, next). For `read-file-name-internal',
these are `completion--embedded-envvar-table' and `completion--file-name-table'.
`read-file-name-internal' doesn't really call directly for any low-level
completion act (e.g. `try-completion') anymore.

> That table may be decomposed into several distinct cases, but that's
> nothing new.  The same was already the case when completing for
> Info-goto-node, for example.

Well, yes and no. Yes, this is a quantitative expansion of the kind of thing
that has already been done in `Info-read-node-name' (but it did only one kind of
completion or another, not a sequence of different kinds of completions) - but
it pretty much becomes a qualitative change because it is so pervasive now.

> > So not only is checking `minibuffer-completion-table' against
> > `read-file-name-internal' useless,
> 
> Comparing functions is usually a bad idea.  Sometimes, 
> there's not much else we can do, admittedly.

The Emacs code still compares `minibuffer-completion-table' against
`read-file-name-internal', in particular, in many locations. I assume that all
of those occurrences still work as intended. ;-)

But I might want to do something different, depending on whether, say, an env
var is being completed or a file name (properly speaking) is being completed at
the moment (or was completed successfully). Maybe just display a msg; it really
doesn't matter what. This doesn't depend on any particular concrete case (need).
It is a general request.

> > How about (at least) recording in some global var the table that is
> > currently being used/tried?
> 
> It's not even clear what that would mean.

Maybe have a global variable `current-completion-table', analogous to
`minibuffer-completion-table. Maybe `complete-with-action' could set it to the
table it is currently using. That might make it possible to know which type of
completion was used successfully.

I obviously don't have a great feel for how to best implement what I'm asking
for or how best to use it.

It just seems that we should be able to have some way to test what's going on -
what kind of completion is being attempted - and what kind of completion was
successful. The act of completion is now a multi-act epic play. It's good to be
able to know which act is in progress or what the outcome of the previous act
was. That's all.

Re: how to determine the current table (really) being used for minibuffer completion?

[Stefan Monnier]

> How I used to do what? There is no concrete example.

Please, then say clearly that your problem is hypothetical, that will
help us answer your questions more directly.

> So you've confirmed what I said: one cannot know. The request is to be
> able to know.  IOW, to be able to have some finer-grain knowledge of
> the completion state.

Define what you mean by "have some finer-grain knownledge of the
completion state".  I think this request is like a solution waiting for
a problem.  So once we find a concrete problem, we'll know better how to
solve it.  But in the abstract like that, it's really unclear what kind
of solution would be needed/possible.

> The point is that now there can be radically different kinds of
> completion that are used, all within the same
> `read-file-name-internal' (for example).

That has always been a possibility.  Maybe this is used a bit more now, tho.

> I would like to be able to know which phase of that completion
> processing I'm in,

When?  Who's "I"?
Do you mean, given the position of point which kind of completion will
take place?  The closest right now is to use completion-boundaries,
which won't tell you what completion table will be used, but at least
it'll tell you which part of the buffer around point will be "affected".

> I would like, for instance (just an example), to be able to distinguish
> file-name completion per se from env var completion, both of which are now
> included as part of "file-name" completion in a larger sense.

completion-boundaries can help you do that.  But again, without knowing to
what purpose you want to distinguish those two cases, it's hard to know
what would really be a good answer.

> Only in a general way. `foo' or `read-file-name-internal' is the
> overall completion function, to be sure, but the completion processing
> is now decomposed into several distinct cases (as you say, next). For
> `read-file-name-internal', these are
> `completion--embedded-envvar-table' and `completion--file-name-table'.
> `read-file-name-internal' doesn't really call directly for any
> low-level completion act (e.g. `try-completion') anymore.

read-file-name-internal is pretty much the only one that I changed in
this way, BTW.  It can now complete either files or env-vars.  In the
future it will probably also be able to complete ~<user> user names.

>> > So not only is checking `minibuffer-completion-table' against
>> > `read-file-name-internal' useless,
>> Comparing functions is usually a bad idea.  Sometimes, 
>> there's not much else we can do, admittedly.
> The Emacs code still compares `minibuffer-completion-table' against
> `read-file-name-internal', in particular, in many locations.

Not sure how you define "many", but my grep found only one place where
we do it: in complete.el which could advise read-file-name-internal
instead (but since it's on the way to obsolescence, it doesn't matter
much either way).

> I assume that all of those occurrences still work as intended. ;-)

Why wouldn't they? ;-)
BTW, all the uses I've ever found (other than complete.el's) were
changed to use minibuffer-completing-file-name or to work in all cases.

> It is a general request.

We have plenty of real concrete problems to address and improvements to
implement, so yours will inevitably end up at the very bottom of
the lot.

> I obviously don't have a great feel for how to best implement what I'm
> asking for or how best to use it.

Come back when you do know.


        Stefan

RE: how to determine the current table (really) being used for minibuffer completion?

[Drew Adams]

Hypothetical example that might make the request clearer:

(completion-all-completions
 STRING 'read-file-name-internal nil (length STRING))

The result returned might be a list of relative file names, or it might be a
list of env vars. A result such as ("CATACOMBS" "CATAPHILE" "CATASTROPHE" . 4)
could be either.

I would like (via Lisp) to know which kind of completion was in fact used
successfully: file-name completion per se or env var completion.

I don't want to analyze STRING (e.g. check for `$') to figure out what might be
the case. (completion-boundaries STRING 'read-file-name-internal nil "") just
gives something like (0 . 0), indicating the whole STRING.

Emacs should be able to tell me directly what the last completion table/function
used was. It should be easy to make this info available somehow, e.g. in a
global variable.

Similarly, for multiple, successive completion types using `completion-styles',
I would like to know which type (style) was in fact successful.

This is the kind of thing I'm talking about.

Re: how to determine the current table (really) being used for minibuffer completion?

[Stefan Monnier]

> Hypothetical example that might make the request clearer:
> (completion-all-completions
>  STRING 'read-file-name-internal nil (length STRING))

> The result returned might be a list of relative file names, or it
> might be a list of env vars.  A result such as ("CATACOMBS"
> "CATAPHILE" "CATASTROPHE" . 4) could be either.

> I would like (via Lisp) to know which kind of completion was in fact used
> successfully: file-name completion per se or env var completion.

In which form?  Do you want a symbol like `file' vs `envvar', or do you
want a completion-table (which can then be a has-table, an obarray, an
alist, a list, a symbol-function, or a lambda expression)?
The second might be doable in many cases (actually more so in Emacs-23
than in Emacs-22, since most functional completion tables now get
composed using primitives like complete-with-action or
completion-table-*, so we could add a 5th method (additionally to the
original 3, try-completions, all-completions, and test-completion, plus
the new completion-boundaries) a bit more easily), but I'm not sure it
would be easy to use.

> I don't want to analyze STRING (e.g. check for `$') to figure out what
> might be the case. (completion-boundaries STRING
> 'read-file-name-internal nil "") just gives something like (0 . 0),
> indicating the whole STRING.

(completion-boundaries "to/to" 'read-file-name-internal nil "") returns
(3 . 0),
(completion-boundaries "to/$to" 'read-file-name-internal nil "") returns
(4 . 0) and 
(completion-boundaries "to/${to" 'read-file-name-internal nil "") returns
(5 . 0), so by looking at (aref STR (1- (car (completion-boundaries STR ...))))
you can "easily" tell which case is currently relevant.  But of course,
that requires knowledge of the behavior of read-file-name-internal.

> Emacs should be able to tell me directly what the last completion
> table/function used was.  It should be easy to make this info
> available somehow, e.g. in a global variable.

It might be easy in 99% of the cases, depending on exactly what
you want.  In my experience, providing this info before seeing actual
uses for it is a perfect recipe for "features" which are unusable in
practice because they don't quite give the info we need.  So I'm
definitely not opposed to providing this info, but I'll wait for
a practical use for it, so I'll be in a better position to provide the
right info.

> Similarly, for multiple, successive completion types using
> `completion-styles', I would like to know which type (style) was in
> fact successful.

Same answer here.  Actually, this one is even a bit worse, because
I strongly suspect that if some code needs this info, it's probably not
doing the right thing.  But in any case, we'll see it when we get there.


        Stefan

RE: how to determine the current table (really) being used for minibuffer completion?

[Drew Adams]

> > Hypothetical example that might make the request clearer:
> > (completion-all-completions
> >  STRING 'read-file-name-internal nil (length STRING))
> 
> > The result returned might be a list of relative file names, or it
> > might be a list of env vars.  A result such as ("CATACOMBS"
> > "CATAPHILE" "CATASTROPHE" . 4) could be either.
> 
> > I would like (via Lisp) to know which kind of completion 
> > was in fact used successfully: file-name completion per se
> > or env var completion.
> 
> In which form?  Do you want a symbol like `file' vs `envvar', 
> or do you want a completion-table (which can then be a
> has-table, an obarray, an alist, a list, a symbol-function,
> or a lambda expression)? The second might be doable in many
> cases (actually more so in Emacs-23 than in Emacs-22, since
> most functional completion tables now get composed using
> primitives like complete-with-action or completion-table-*,
> so we could add a 5th method (additionally to the original 3,
> try-completions, all-completions, and test-completion, plus
> the new completion-boundaries) a bit more easily), but I'm
> not sure it would be easy to use.

As you recognize, it is better to test a known symbol (e.g. `envvar') than a
completion table (function or list or hash table or obarray...).

What I suggest is this:

Use a data structure (an alist), to have a standard list of symbols (the keys of
the alist) that can be tested.

@ In the case of completion styles (determining which style performed the
completion), just reuse `completion-styles-alist': be able to test the alist key
- `basic', `emacs22', `emacs21', or `partial-completion'. (But using whatever
real alist entries are in `completion-styles-alist' at the time of testing.)
This will provide a knowable and unambiguous set of symbols to test.

@ In the case of env-var and file-name completion: Do the same thing. Implement
something similar to `completion-styles-alist'. It would be good to use an alist
here anyway, for flexibility of use and ease of maintenance.

After successful completion, set a global variable to the symbol that represents
the completion style.

This is simply additional information to be, in effect, returned by the
completion function. We already return the set of completions, and you've hacked
that list to also carry the base size in its last cdr. This third piece of info
about the completion state can be returned in a global variable. (It can be at
least as important as the base size, IMO.)

Your code comments acknowledge the hackiness of how base-size is returned, and
they point to possibly also returning a `completion-extra-size' and
`completion-no-auto-exit' in the future. The point is that completion is a
complex process, and it can return more useful info than simply the list of
completions.

--

Also, it is somewhat inconvenient to have `completion-styles-alist' as a defvar,
not a defcustom. It is the real fulcrum for all of the code. It is baked once by
Emacs development, and then it is used as the basis for the rest. User
customization of `completion-styles' is limited to choosing among predefined
styles.

It would be better to have a single option that lets users both (a) choose the
styles to use and (b) define new styles (their names and their defining
completion functions). IOW, combine `completion-styles-alist' with
`completion-styles' as a single option that users can customize.

That way, the code would still automatically adjust to the possible set of
completion styles, but those styles would not be limited to choices from a
predefined list. Users could still just as easily add or remove predefined
styles, but they could also more easily add their own.

> > Emacs should be able to tell me directly what the last completion
> > table/function used was.  It should be easy to make this info
> > available somehow, e.g. in a global variable.
> 
> It might be easy in 99% of the cases, depending on exactly what
> you want.

See above.

Re: how to determine the current table (really) being used for minibuffer completion?

[Stefan Monnier]

> After successful completion, set a global variable to the symbol that
> represents the completion style.

Returning info via a global variable is evil.

> We already return the set of completions, and you've hacked that list
> to also carry the base size in its last cdr.

No, that was an ugly hack I managed to get rid of before releasing
Emacs-23.

> This third piece of info about the completion state can be returned in
> a global variable. (It can be at least as important as the base
> size, IMO.)

Reality check: you can't even come up with a concrete case where it
would be useful, so how can you claim it can be at least as important as
the base-size which has been needed since the introduction of
choose-completion (Emacs-21?  Emacs-20?)

> Also, it is somewhat inconvenient to have `completion-styles-alist' as
> a defvar, not a defcustom. It is the real fulcrum for all of the
> code. It is baked once by Emacs development, and then it is used as
> the basis for the rest. User customization of `completion-styles' is
> limited to choosing among predefined styles.

A defcustom?  It contains pointers to functions and those functions use
a non-trivial calling convention, so there's very little chance you'll
be able to add entries there without writing Lisp code.
`completion-styles-alist' is definitely not a candidate for defcustom.

> It would be better to have a single option that lets users both (a)
> choose the styles to use and (b) define new styles (their names and
> their defining completion functions). IOW, combine
> `completion-styles-alist' with `completion-styles' as a single option
> that users can customize.

And where do you keep the list of options from which they can choose?

> That way, the code would still automatically adjust to the possible set of
> completion styles, but those styles would not be limited to choices from a
> predefined list. Users could still just as easily add or remove predefined
> styles, but they could also more easily add their own.

How hard can it be to add their new style to completion-styles-alist and
then to completion-styles as well?  Especially compared to the amount of
work required in writing that new style?

I must be misunderstanding something.

>> It might be easy in 99% of the cases, depending on exactly what
>> you want.
> See above.

The "above" didn't tell me what you want to do.


        Stefan


PS: FWIW, I can think of at least one case where the "file vs envvar"
information could potentially be used.  So maybe if you try hard enough,
you too can come up with an example.