bug#73046: 29.4; Emacs 100% CPU usage for several seconds when opening dired buffer over TRAMP

[Eli Zaretskii <eliz@gnu.org>]

> From: Suhail Singh <suhailsingh247@gmail.com>
> Cc: Suhail Singh <suhailsingh247@gmail.com>,  michael.albinus@gmx.de,
>   73046@debbugs.gnu.org
> Date: Fri, 06 Sep 2024 20:19:34 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > It needs to show around 40KB to explain 10 sec of delay.
> 
> I don't understand your reasoning.

It's simple arithmetics: if fetching a 20KB file takes 4 to 5 sec,
then the 10-sec delay you reported for Dired should mean we are
fetching about 40KB of data.

> In fact if the output of ls -al was indeed around 40kb I would have
> been very surprised.

In my home directory I get:

 $ ls -al ~ | wc
     708    6371   43324

So a large enough (but not outlandishly large: just 700 files)
directory can indeed mean Emacs needs to fetch 40KB of data for
refreshing a Dired buffer.  Which led me to believe this is not an
impossible situation.

> The time taken for transferring the 20KB file included initial
> connection costs whereas TRAMP would presumably be reusing the same
> connection, but would be sending multiple small requests.

This just makes my argument stronger: it would mean that even 40KB
data moved for Dired would not quite explain the 10-sec delay.

> I did some further investigation; summarizing findings below:
> 
> A. On Host-A, the network connection is fairly slow s.t. transferring a
>    20KB file takes ~5s.  On Host-B, the network connection is fairly
>    fast.
> 
> B. On Host-A, the time taken to refresh dired buffer containing 22
>    Subdirectories (/tmp/test/src as in above code snippet) is 0.70-0.75s
>    with font-lock enabled, and about the same with font-lock disabled.
>    These times exclude the time taken to establish the intiial
>    connection over TRAMP.
> 
> C. On Host-A, the time taken to refresh dired buffer containing 22
>    symlinks (each symlink pointing to a directory, i.e., /tmp/test/links
>    in the above code snippet) is 0.70-0.75s with font-lock disabled.
>    With font-lock enabled the time taken is ~13-14s and the CPU is at
>    100%.  These times exclude the time taken to establish the intiial
>    connection over TRAMP.
> 
> D. On Host-B, the time taken to display dired buffer for /tmp/test/links
>    with font-lock enabled is ~2s greater than when font-lock is
>    disabled.  When /tmp/test/links contains 100 symlinks to directories
>    (instead of 22), the time taken when font-lock is enabled is ~6s
>    greater than when font-lock is disabled.
> 
> Given above, I conclude:
> 
> 1. The issue is present when there are symlinks to directories.
> 
> 2. The issue is worse when there are greater number of symlinks to
>    directories.
> 
> 2. The issue is worse when the connection is slower.  However, it is
>    still observable when the connection is faster - if you're having
>    difficulty reproducing, increase the number of symlinks to
>    directories in /tmp/test/links above.
> 
> 3. Given that when connection is slower, not only is the time taken for
>    font-locking greater, but the CPU is at 100%, I suspect that the
>    relevant code is doing some kind of busy-waiting.

Thanks you.  So the problem seems to be symlinks, and specifically
symlinks to directories.  Michael, what does Tramp do specially in
these cases that could explain the slowdown?

> The above observations seem consistent with Michael's comments above
> regd. font-lock checks for "Broken Symbolink link" and "Symbolic link to
> a directory".

Michael, what do these checks entail, and why are they so
CPU-expensive and take a lot of time with slow connections?