bug#41423: 27.0.91; eshell file completion in tramp dir is slow (3 minutes) [regression on pretest]

[Stefan Monnier <monnier@iro.umontreal.ca>]

> I do not understand why I should explain to you how the code you wrote
> works.

[ Because I remember ow it's supposed to work, but I don't know how it
  actually behaves in this specific case.  ]

> 19. given that the value of the start position did not change, the lambda
>     let-bound at step 8 returns t, and therefore completion-in-region--postch
>     does not exit completion-in-region-mode
> 20. completion-in-region--postch is now finished, it did not change
>     anything in the eshell buffer
> 21. RET is pressed
> 22. post-command-hook is executed, and still contains
>     completion-in-region--postch, so it is called again
> 23. completion-in-region--postch calls completion-in-region-mode-predicate
> 24. this calls pcomplete-completions-at-point a third time, which calls
>     pcomplete-completions

Looks OK so far.

> 25. for some reason, pcomplete-completions considers that it must now
>     complete a command name and not a directory name

I guess this is because after RET we're now at BOL so it looks like
a brand new command is starting.

> 26. therefore pcomplete-completions does not call pcomplete/cd but
>     eshell-complete-commands-list

I guess this is the culprit and this is where the time is spent.

`eshell-complete-commands-list` eagerly builds the list of possible
candidates and it takes a while whereas we should here return something
quickly and cheaply, e.g. by returning a function which will build and
return this list more lazily when completion is actually performed.

Of course, the slowdown will presumably still be seen when you do
actually want to complete a command name, so we should probably try and
figure out more precisely where the slowdown comes from and how to
avoid/reduce it.

I guess part of the slowdown comes from the fact that we don't just use
`file-name-all-completions` in each directory in PATH but we
additionally call `file-executable-p` (or `file-readable-p`) on every
command found, which I expect will take quite a while when it goes
through Tramp.

Still, I'm not completely sure where the time is spent because I'm not
sure which files/dirs will go through tramp.  AFAICT, `eshell-get-path`
will return the "local" $PATH rather than that of the remote host ... oh
wait, no I see that `tramp-eshell-directory-change` will set that to the
remote host's $PATH, so it should indeed work correctly (but slowly).

Maybe `tramp-eshell-directory-change` should tell
`eshell-complete-commands-list` to cache the list and also not to bother
checking `file-executable-p`?

> And the last steps should be:
>
> 29. when eshell-complete-commands-list has finished its job,
> pcomplete-completions-at-point returns a value in which the start position
> has changed
> 30. therefore the lambda let-bound at step 8 returns nil, and therefore
> completion-in-region--postch exits completion-in-region-mode entered at step
> 10
> 31. this removes completion-in-region--postch from post-command-hook
> 32. eshell finally prints its next prompt

Yes, these steps look fine.


        Stefan