bug#56342: TRAMP (sh) issues way too many commands, thus being very slow over high-ping networks

[Michael Albinus <michael.albinus@gmx.de>]

Paul Pogonyshev <pogonyshev@gmail.com> writes:

Hi Paul,

> Understood, but I propose to adopt a different benchmark: number of
> remote commands.  As soon as you are not in a LAN, even not when using
> a particularly slow network, this becomes an order or two of magnitude
> more important than everything else.  E.g. with slightly slower
> commands or particularly inefficient Elisp you can make it 2-10 ms
> slower for everyone.  But with unoptimized command flow (i.e. more
> remote commands) it can be 100-500 ms slower, though not for everyone,
> but people using this over non-LAN.  I think in such cases extremes
> are more important than the average.

I agree, sending something over the wire is always slower than computing
clever caches locally.

>> And directories are also problematic for caches. As soon as something
>> changes there (creation or deletion of a file, for example), the cached
>> values of the directory must be flushed.
>
> Yeah, I suppose Tramp has no generic way to know that Magit issues
> reading commands.  Can we devise a generic interface that would tell
> Tramp: "Commands within this block will not modify file system",
> e.g. with let-binding some variable?

They will modify the file system, although sometimes only the
timestamps.

Until now we have only something similar for synchronous
processes: process-file-side-effects. I have no idea whether package
authors are aware of this, and let-bind it to nil in case of. In the
magit sources, I haven't found this variable.

> In general, it feels like Tramp flushes its caches too often or maybe
> doesn't even cache certain things at all.  I.e. it's not about those
> 10 seconds (following your advice I have even increased that).  It's
> that while serving one user-level command here (i.e. within 3 seconds
> at most), it can issue the same remote command several times, thus
> not reusing previous results.

Well, that's right. If, for example, the file modes are changed, Tramp
flushes all caches for that file, although some of them
("file-exists-p", "file-directory-p" etc) aren't affected.

> E.g. in the traces you have attached this can be seen.  The following
> two commands repeat twice:

I haven't investigated this special case yet, but last days I'm working
on exact this problem. Flushing caches shouldn't be a sledge hammer, but
fine grained, selecting the needed properties to be flushed. Let's see
where we land.

> Paul

Best regards, Michael.