bug#41423: 27.0.91; eshell file completion in tramp dir is slow (3 minutes) [regression on pretest]

[Gregory Heytings via "Bug reports for GNU Emacs, the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>]

>
> Hmm... any hope you could refine your description accordingly?
>

I do not understand why I should explain to you how the code you wrote 
works.  Anyway, here we go (and I fear you will now tell me that my 
description is now too refined):

1. start emacs -Q
2. in an eshell buffer, type "<command> <first letters of a directory name> TAB" (command can be "cd", "ls", "rm", ...)
3. TAB calls completion-at-point
4. completion-at-point looks at completion-at-point-functions, whose value is (pcomplete-completions-at-point t), and calls pcomplete-completions-at-point
5. pcomplete-completions-at-point calls pcomplete-completions, which calls pcomplete/cd, which calls pcomplete--here, ...
6. pcomplete-completions returns a collection of completion candidates, and pcomplete-completions-at-point returns that collection together with a function pointer (to pcomplete-completions-at-point itself), a start position, an end position, and a property list
7. we are now back in completion-at-point, and enter the second case in its pcase
8. completion-at-point let-binds completion-in-region-mode-predicate to a lambda, which calls pcomplete-completion-at-point
9. completion-at-point then calls completion-in-region, which calls completion--in-region
10. completion--in-region enters completion-in-region-mode, which adds completion-in-region--postch to post-command-hook
11. completion--in-region calls completion--in-region-1, which calls completion--do-completion, which finally does the actual completion based on the collection of completion candidates returned at step 6
12. at this point the eshell buffer contains the completed directory name, with a trailing slash
13. completion-at-point is now finished, and post-command-hook is executed
14. post-command-hook calls completion-in-region--postch
15. completion-in-region--postch calls completion-in-region-mode-predicate (in fact, completion-in-region-mode--predicate which has been set to completion-in-region-mode-predicate when entering completion-in-region-mode at step 10)
16. this calls pcomplete-completions-at-point a second time, which calls pcomplete-completions, which calls pcomplete/cd, which calls pcomplete--here, ...
17. pcomplete/cd and pcomplete-completions return the exact same values as in step 6
18. pcomplete-completions-at-point returns almost the same value as in step 6 (the only difference is the value of the end position)
19. given that the value of the start position did not change, the lambda let-bound at step 8 returns t, and therefore completion-in-region--postch exits completion-in-region-mode entered at step 10
20. completion-in-region--postch is now finished, it did not change anything in the eshell buffer
21. RET is pressed
22. post-command-hook is executed, and still contains completion-in-region--postch, so it is called again
23. completion-in-region--postch calls completion-in-region-mode-predicate
24. this calls pcomplete-completions-at-point a third time, which calls pcomplete-completions
25. for some reason, pcomplete-completions considers that it must now complete a command name and not a directory name
26. therefore pcomplete-completions does not call pcomplete/cd but eshell-complete-commands-list
27. eshell-complete-commands-list loops through all possible command names 
28. if these command names had a common prefix, it would have been inserted in the eshell buffer (?), but this is not the case, so the effect of this loop (apart from a waste of time) is nil
29. when eshell-complete-commands-list has finished its job, eshell prints its next prompt

At step 27 it is possible to interrupt the loop with C-g.  At step 21 it 
is possible to remove completion-in-region--postch from post-command-hook, 
for example by switching buffers with C-x C-b.

>
> Calling `pcomplete-completions-at-point` several times is not 
> necessarily a problem.  E.g. it's normal to call it a second time after 
> completion to check whether we're still in the same completion area (in 
> order to detect when completion is "finished").
>

For some general case , I don't know (I admit I can't think of a case in 
which this would be useful, but I know my experience is limited).  To 
complete a directory name in a shell, I don't see why this should be the 
case.  The (now obsolete) mechanism calls pcomplete (which also calls 
pcomplete/cd) a single time, and it worked.  I don't see what steps 13-29 
could possibly do that would be useful, at least in the context of a 
shell.