bug#41423: 27.0.91; eshell file completion in tramp dir is slow (3 minutes) [regression on pretest]

[Gregory Heytings via "Bug reports for GNU Emacs, the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>]

Hi Stefan,

Thank you very much for your detailed answer!

>
>>> This design relies on the fact that completion tables can be lazy, so 
>>> it should always be possible to make the completion-at-point-function 
>>> very cheap and harmless, so it's OK to call it repeatedly (or even 
>>> needlessly).
>>
>> This is not at all documented AFAICS.  Given that it's a crucial aspect 
>> for your design to work, it should be.
>
> The lispref says the following:
>
>    The functions on this hook should generally return quickly, since
>    they may be called very often (e.g., from @code{post-command-hook}).
>    Supplying a function for @var{collection} is strongly recommended if
>    generating the list of completions is an expensive operation.  Emacs
>    may internally call functions in @code{completion-at-point-functions}
>    many times, but care about the value of @var{collection} for only
>    some of these calls.  By supplying a function for @var{collection},
>    Emacs can defer generating completions until necessary.  You can use
>    @code{completion-table-dynamic} to create a wrapper function:
>

I see, thanks for the pointer.  I did not find this because I was 
searching for `pcomplete-completions-at-point' and 
`pcomplete-completions'.  It would make sense to put this pointer in 
pcomplete.el.

>
> W.r.t simple mechanism for simple cases, I'm not sure what that would 
> concretely look like in this particular case.
>

In this particular case, in Emacs 24-26, `eshell-pcomplete' called 
`pcomplete', which did the completion without triggering the machinery of 
`pcomplete-completion-at-point', that is, without entering a transient 
`completion-in-region-mode', without modifying `post-command-hook', and so 
forth.  In particular, `pcomplete/cd' was called a single time.  It seems 
to me (even now that I understand the design of 
`pcomplete-completion-at-point', and that I understand how the more 
complex mechanism can be made as efficient as the simple one) that this 
simple mechanism is often sufficient, that it is easier to understand, and 
that it should remain available.  OTOH I also understand the point that 
you want to avoid duplicating code.

>
> Some motivations for `pcomplete-completions-at-point`:
> - make it possible to remove duplicate code that deals with the UI aspect of completion (i.e. the `pcomplete` command) rather than the core purpose of `pcomplete.el` which is to provide a way to specify which completion table applies where on a command line.
> - let the `pcomplete` machinery work with the standard UI, which means it can also (mostly) obey `completion-styles`.
> - let the `pcomplete` machinery work with other UIs such as `company-mode`.  I believe this last point is more important now.
>

I see.  This makes perfect sense, thanks for the clarification.

What is still missing IMO is a general description/documentation of the 
various parts of the completion mechanisms (completion-at-point, 
completion-in-region, pcomplete, pcomplete-completion-at-point, 
comint-completion-at-point, icomplete, ...) in Emacs behave and interact. 
I was completely lost when I started working on this bug, things are a bit 
clearer now, but still not very clear.

>
>> Indeed this patch works.
>
> Thanks.
>
[...]
>
> In the patch I sent I think there's a bug in that a leading * should 
> change the START..END returned by 
> `pcomplete-completion-at-point-function` so the `glob-name` computation 
> should be done outside of the `completion-table-dynamic`.
>

*sigh* So this bug can still not be considered fixed?