Recommendation for CAPF setup when you don't know completion string in advance

Recommendation for CAPF setup when you don't know completion string in advance

[JD Smith]

[-- Attachment #1: Type: text/plain, Size: 1593 bytes --]

I’m endeavoring to more fully link completion in python-shell-mode directly to those provided by iPython, when it is running as the inferior shell.  Modern versions of iPython have fantastic built-in completion capabilities when run in a terminal, but they are not well exploited by python.el.  

Here’s the logical flow I’m trying to achieve:

Take the entire line from point back to the iPython prompt (call it `line’), and hand it to iPython’s internal completer:  get_ipython().Completer.complete(line_buffer=line,cursor_pos=len(line)): 
This returns both the sub-text iPython has chosen to complete (a trailing portion of the line), and a list of completions.  
Use this sub-text length to set the `beg’ and `end’ parameters that a CAPF needs to return, and the returned completions list as the `collection’.

In principle I could just do this directly in the CAPF, i.e. interact with the iPython process, compute `beg’ and `end’, return them with the completion list as the collection. But as Programmed Completions node says: 

Supplying a function for collection is strongly recommended if generating the list of completions is an expensive operation. Emacs may internally call functions in completion-at-point-functions many times, but care about the value of collection for only some of these calls.

So it’s a quandary: I won’t yet know `beg’ and `end’ until _after_ interacting with iPython.  Is there any way to “revise” ‘beg’ and ‘end’ in a collection function returned from a CAPF?  If not, other ideas I could try?

Thanks.

[-- Attachment #2: Type: text/html, Size: 2376 bytes --]

Re: Recommendation for CAPF setup when you don't know completion string in advance

[Stefan Monnier]

> So it’s a quandary: I won’t yet know `beg’ and `end’ until _after_
> interacting with iPython.  Is there any way to “revise” ‘beg’ and ‘end’ in
> a collection function returned from a CAPF?

Indeed, that's a problem.
You might be able to get away with the following:

Make your CAPF function return a beg..end that covers "the whole line"
and which returns a completion table in the form of a function.
That function will then defer to the iPython code for the grunt of its
work and will return the "real" boundaries via the
`completion-boundaries` method.  This will probably require some caching
in the completion table so we don't call iPython too many times for
a single completion (like once for `completion-boundaries`, once for
`try-completion`, once for `all-completions`, etc...).

The completion is written under the assumption that
`completion-boundaries` is cheaper to perform than `all-completions`,
so there's a risk it won't work very well, but my intuition tells me it
should work "well enough".


        Stefan

Re: Recommendation for CAPF setup when you don't know completion string in advance

[JD Smith]

[-- Attachment #1: Type: text/plain, Size: 2337 bytes --]



> On Apr 3, 2021, at 7:49 PM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> 
>> So it’s a quandary: I won’t yet know `beg’ and `end’ until _after_
>> interacting with iPython.  Is there any way to “revise” ‘beg’ and ‘end’ in
>> a collection function returned from a CAPF?
> 
> Make your CAPF function return a beg..end that covers "the whole line"
> and which returns a completion table in the form of a function.
> That function will then defer to the iPython code for the grunt of its
> work and will return the "real" boundaries via the
> `completion-boundaries` method.  This will probably require some caching
> in the completion table so we don't call iPython too many times for
> a single completion (like once for `completion-boundaries`, once for
> `try-completion`, once for `all-completions`, etc…).

Thank you, that sounds quite promising.  I’ve started sketching out a mash-up of `completion-table-with-cache’ and `python-shell-completion-at-point’ and it seems it should work out.  

There’s one thing I’m unclear on: how long does the completion engine in Emacs hold on to a collection aka "completion table" function?  I’d expect this function to live only throughout a single “event" of completion (M-tab or what have you).  But if that is indeed the case, I’m confused by why `completion-table-with-cache’ needs to test:

	(string-prefix-p last-arg arg ignore-case)

Won’t this _always_ be true for the custom-wrapped closure which `completion-table-with-cache’ constructs?  In other words, I can see two possibilities:

The cache is there simply to prevent the multiple subsequent collection function calls with the various action flags '(lambda, t, nil, boundaries, metadata) from repeating the expensive completion results lookup.
The cache must guard against _future_ calls of the collection function providing yet other not-yet-conceived-of completion strings.  

If #1 is true, wouldn’t all of these calls provide the same completion string? And if #2 is true, why does the completion engine “hold on” to a bespoke completion table function beyond the initial completion event which created it?  

Probably I’m missing the real #3.  This is relevant here for deciding when to re-consult IPython for new completion data. 


[-- Attachment #2: Type: text/html, Size: 4393 bytes --]

Re: Recommendation for CAPF setup when you don't know completion string in advance

[Stefan Monnier]

> There’s one thing I’m unclear on: how long does the completion engine in
> Emacs hold on to a collection aka "completion table" function?

As long as needed.  It all depends on the situation.
E.g. if the completion table is passed to `completing-read`, then it
will be used during the whole lifetime of the minibuffer.

When the completion table comes from CAPF, it's usually used for a much
shorter duration, among other things because we have to regularly call
CAPF in order to detect when changes to the buffer's content or to
`point` have caused a change in the kind of things we're completing.

> I’d expect this function to live only throughout a single “event" of
> completion (M-tab or what have you).

It will often live a little longer, but not much, usually, indeed.
This said, it all depends on the UI.  I can imagine a UI where CAPF is
called once and then the completion is performed within a kind of modal
interface which insures that CAPF doesn't need to be re-called, in which
case the completion-table may be used for quite a while, just as is the
case in minibuffers.

> I’m confused by why `completion-table-with-cache’ needs to test:
>
> 	(string-prefix-p last-arg arg ignore-case)

This was designef for completion-tables in general (rather than only
those used in a particular kind of situation).  Furthermore, even
a single M-TAB completion may easily query the completion table several
times with different prefix strings (because of `completion-styles`).

> Won’t this _always_ be true for the custom-wrapped closure which
> `completion-table-with-cache’ constructs?

I'm pretty sure there will be cases where this is not the case, no.

E.g. you may have a first call for the `basic` completion style with
a prefix "foo" which ends up returning nothing, after which we fall back
to `substring` completion style and hence query the completion table
again with a prefix of "".


        Stefan

Re: Recommendation for CAPF setup when you don't know completion string in advance

[JD Smith]

[-- Attachment #1: Type: text/plain, Size: 945 bytes --]



> On Apr 3, 2021, at 11:20 PM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> 
>> I’m confused by why `completion-table-with-cache’ needs to test:
>> 
>> 	(string-prefix-p last-arg arg ignore-case)
> 
> This was designef for completion-tables in general (rather than only
> those used in a particular kind of situation).  Furthermore, even
> a single M-TAB completion may easily query the completion table several
> times with different prefix strings (because of `completion-styles`).


Thanks, I get it.  Since iPython’s suggestions will never change for the same line of input, perhaps caching them “once and forever” in a given collection function will be OK.  This presumes of course that none of the possible cases for long term use of a collection function will operate on distinct lines of input.  I need to work out some issues on iPython’s completer backend; will revisit once that’s working well.


[-- Attachment #2: Type: text/html, Size: 1950 bytes --]

Re: Recommendation for CAPF setup when you don't know completion string in advance

[JD Smith]

> On Apr 3, 2021, at 7:49 PM, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
> 
>> So it’s a quandary: I won’t yet know `beg’ and `end’ until _after_
>> interacting with iPython.  Is there any way to “revise” ‘beg’ and ‘end’ in
>> a collection function returned from a CAPF?
> 
> Indeed, that's a problem.
> You might be able to get away with the following:
> 
> Make your CAPF function return a beg..end that covers "the whole line"
> and which returns a completion table in the form of a function.
> That function will then defer to the iPython code for the grunt of its
> work and will return the "real" boundaries via the
> `completion-boundaries` method.  This will probably require some caching
> in the completion table so we don't call iPython too many times for
> a single completion (like once for `completion-boundaries`, once for
> `try-completion`, once for `all-completions`, etc...).


Thanks for these good suggestions.  I have been attempting a solution along these lines, and I think the performance is fine.  I notice that CAPF first asks for metadata (which are static here), then for the boundaries.  At that point, I don’t know the boundaries, so I talk to the iPython process, and ask it for all the completions for the entire line, given my position within it. It returns these, along with the portion of the full line it has decided to complete.  From this, I compute and return the boundaries.  Then, in subsequent calls to my table lambda, CAPF proceeds with its various ACTION's.  

Pseudo-code for the completion table function I’m building:

(lambda (string pred action)
  (when (no-result-yet-cached-or-cached-string-is-substring-of-string)
    (if (eq action 'metadata) `(metadata …)
      (unless last-prefix
	;; talk to ipython and save the completion info
	)
      (if (eq (car-safe action) 'boundaries) ; munge the boundaries
	  `(boundaries from . to) ; using saved data from ipython
	(complete-with-action action completion-alist string pred)))))

The problem I have encountered is in the final call to `complete-with-action’.  The original string may be an entire line, such as:

	for i in ran[Tab]

and iPython correctly figures out to complete just a portion of that string; say ran -> range.  I tell CAPF about these boundaries (in this case, 9 . 0).  If, however, I later call complete-with-action (which just calls try-completion, all-completions, etc.) with the entire line string (“for in ran”), together with a completion-alist that contains iPython’s sub-string completions (like “range”), it thinks there is no completion.  If instead I just peel off the part of the full line that needs completing (“ran”) and pass that to complete-with-action as STRING, it recognizes that there's a good completion now, but then _replaces the entire line_ with the result.

How do I get complete-with-action/try-completion/test-completion/all-completions etc. to respect my boundaries?

Re: Recommendation for CAPF setup when you don't know completion string in advance

[Stefan Monnier]

> (lambda (string pred action)
>   (when (no-result-yet-cached-or-cached-string-is-substring-of-string)
>     (if (eq action 'metadata) `(metadata …)
>       (unless last-prefix
> 	;; talk to ipython and save the completion info
> 	)
>       (if (eq (car-safe action) 'boundaries) ; munge the boundaries
> 	  `(boundaries from . to) ; using saved data from ipython
> 	(complete-with-action action completion-alist string pred)))))
[...]
> and iPython correctly figures out to complete just a portion of that
>  string; say ran -> range.  I tell CAPF about these boundaries (in
>  this case, 9 . 0).  If, however, I later call complete-with-action
>  (which just calls try-completion, all-completions, etc.) with the
>  entire line string (“for in ran”), together with a completion-alist
>  that contains iPython’s sub-string completions (like “range”), it
>  thinks there is no completion.  If instead I just peel off the part
>  of the full line that needs completing (“ran”) and pass that to
>  complete-with-action as STRING, it recognizes that there's a good
>  completion now, but then _replaces the entire line_ with the result.
>
> How do I get
> complete-with-action/try-completion/test-completion/all-completions
> etc. to respect my boundaries?

There's `completion-table-with-context`.
Note that it comes with the following:

    ;; TODO: add `suffix' maybe?

so you probably won't be able to use it as-is and you'll need to write
your function instead.  But hopefully it should be enough to get
you started.  It shouldn't be terribly hard.

I think one problem you may encounter is that in the default UI,
selecting an entry from *Completions* (e.g. with a middle click) will
probably throw away the "suffix" (i.e. the text after the end of the
boundary).
If so, I suggest you `M-x report-emacs-bug` (tho maybe I did fix this
one already; but there might be a few other such corner cases where we
throw away the text after the end of the boundary).


        Stefan

Re: Recommendation for CAPF setup when you don't know completion string in advance

[JD Smith]

[-- Attachment #1: Type: text/plain, Size: 1333 bytes --]

> 
> There's `completion-table-with-context`.
> Note that it comes with the following:
> 
>    ;; TODO: add `suffix' maybe?
> 
> so you probably won't be able to use it as-is and you'll need to write
> your function instead.  But hopefully it should be enough to get
> you started. 

Looking at `completion-table-with-context’, it seems just to prepend the prefix after the completion happens, if a string is returned from `complete-with-action'.  I’m struggling then to understand the point of completion-boundaries lookup at all.  I presume the boundaries returned by a table function are relative to the string enclosed by then original START…END range originally returned by the CAPF.  

To make this concrete, suppose you are completing this string:

/path/to/some[Tab]

and your CAPF indicates the range of the entire string, but then its table function returns non-trivial boundaries: `(boundaries 9 . 0).  In this case, the `completion-file-name-table’ still also trims the full path string to its non-directory end component (‘some’) before passing on to `file-name-completion`.  Then finally it prepends the directory back on again. So what good did returning boundaries do it, if it had to cut out the completion string of interest, find completions for it, then glue the prefix back on anyway?

[-- Attachment #2: Type: text/html, Size: 6832 bytes --]

Re: Recommendation for CAPF setup when you don't know completion string in advance

[Stefan Monnier]

> Looking at `completion-table-with-context’, it seems just to prepend the
> prefix after the completion happens, if a string is returned from
> `complete-with-action'.

Indeed.

> I’m struggling then to understand the point of
> completion-boundaries lookup at all.

That's catering to more complex situations.
E.g. imagine completing

    foo:file/name:bar

where your code split this up into a `foo:` prefix and a `:bar` suffix,
and it uses the `completion-file-name-table` as completion table for the
"middle part".

In order to compute the overall boundaries to use when point is inside
"name", you need to add the length of the prefix to the boundaries
returned by `completion-file-name-table`.

IIUC in your case the completion-table used between prefix and suffix
will always be a "plain list" with no "nested" boundaries so you won't
need to worry about that.


        Stefan

Re: Recommendation for CAPF setup when you don't know completion string in advance

[Daniel Mendler]

On 5/11/21 6:04 AM, Stefan Monnier wrote:
> There's `completion-table-with-context`.
> Note that it comes with the following:
> 
>     ;; TODO: add `suffix' maybe?
> 
> so you probably won't be able to use it as-is and you'll need to write
> your function instead.  But hopefully it should be enough to get
> you started.  It shouldn't be terribly hard.
> 
> I think one problem you may encounter is that in the default UI,
> selecting an entry from *Completions* (e.g. with a middle click) will
> probably throw away the "suffix" (i.e. the text after the end of the
> boundary).
> If so, I suggest you `M-x report-emacs-bug` (tho maybe I did fix this
> one already; but there might be a few other such corner cases where we
> throw away the text after the end of the boundary).

I didn't follow the discussion here, but there is one thing I noticed -
the throw away of the suffix.

In my Vertico and Corfu packages I have the following problematic
behavior, which is also present in the default completion (I copied the
comment verbatim from vertico.el):

;; XXX There is a small bug here, depending on interpretation. When
;; completing "~/emacs/master/li|/calc" where "|" is the cursor,
;; then the returned candidate only includes the prefix
;; "~/emacs/master/lisp/", but not the suffix "/calc". Default
;; completion has the same problem when selecting in the
;; *Completions* buffer.

Is it non-intentional that the suffix is thrown away? I accidentally
replicated this behavior in Vertico and Corfu, but wasn't sure if this
is how it is actually meant to be.

Daniel

Re: Recommendation for CAPF setup when you don't know completion string in advance

[Stefan Monnier]

> ;; XXX There is a small bug here, depending on interpretation. When
> ;; completing "~/emacs/master/li|/calc" where "|" is the cursor,
> ;; then the returned candidate only includes the prefix
> ;; "~/emacs/master/lisp/", but not the suffix "/calc". Default
> ;; completion has the same problem when selecting in the
> ;; *Completions* buffer.
>
> Is it non-intentional that the suffix is thrown away?

I think the answer depends on the completion-style.  IIRC some
completion styles do that on purpose (in order to preserve/reproduce
the behavior of earlier completion code), whereas I seem to remember
a few cases where we may do something like that not on purpose (rather
because of a bug, or out of laziness).


        Stefan

Re: Recommendation for CAPF setup when you don't know completion string in advance

[Daniel Mendler]

On 5/11/21 5:57 PM, Stefan Monnier wrote:
>> ;; XXX There is a small bug here, depending on interpretation. When
>> ;; completing "~/emacs/master/li|/calc" where "|" is the cursor,
>> ;; then the returned candidate only includes the prefix
>> ;; "~/emacs/master/lisp/", but not the suffix "/calc". Default
>> ;; completion has the same problem when selecting in the
>> ;; *Completions* buffer.
>>
>> Is it non-intentional that the suffix is thrown away?
> 
> I think the answer depends on the completion-style.  IIRC some
> completion styles do that on purpose (in order to preserve/reproduce
> the behavior of earlier completion code), whereas I seem to remember
> a few cases where we may do something like that not on purpose (rather
> because of a bug, or out of laziness).

I see. There is some code in `choose-completion-string` which checks if
the length of the resulting string equals the car of the
`completion-boundaries`. When choosing a directory, the minibuffer is
not automatically exited. But the suffix string after the directory is
still thrown away. I don't see where the completion style is involved here.

The problem is probably that when appending the suffix the resulting
choice is not necessarily a valid candidate. This is at least how I
thought about it when replicating the behavior in Vertico/Corfu.

We could adjust the `choose-completion` behavior such that the suffix is
never thrown away and completion is never automatically exited when a
suffix exists. Would that make sense or did I miss something? Is this
worth a bug report/patch?

Daniel

Re: Recommendation for CAPF setup when you don't know completion string in advance

[Stefan Monnier]

>> I think the answer depends on the completion-style.  IIRC some
>> completion styles do that on purpose (in order to preserve/reproduce
>> the behavior of earlier completion code), whereas I seem to remember
>> a few cases where we may do something like that not on purpose (rather
>> because of a bug, or out of laziness).
>
> I see. There is some code in `choose-completion-string` which checks if
> the length of the resulting string equals the car of the
> `completion-boundaries`. When choosing a directory, the minibuffer is
> not automatically exited. But the suffix string after the directory is
> still thrown away. I don't see where the completion style is involved here.

Sounds like a bug then.

> The problem is probably that when appending the suffix the resulting
> choice is not necessarily a valid candidate. This is at least how I
> thought about it when replicating the behavior in Vertico/Corfu.

Something like that, yes.

> We could adjust the `choose-completion` behavior such that the suffix is
> never thrown away and completion is never automatically exited when a
> suffix exists. Would that make sense or did I miss something?

I think it makes sense.

> Is this worth a bug report/patch?

Yes.


        Stefan