bug#56342: TRAMP (sh) issues way too many commands, thus being very slow over high-ping networks

[Paul Pogonyshev <pogonyshev@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 4420 bytes --]

Some more thoughts. Why does it even need `echo are you awake'? It's a
network connection, it can still fail even if it worked fine 1 ms before
when you checked. So, why not just let the first command fail if the
connection is dead and restart the connection if it fails in such a way as
to suspect that it is dead (i.e. no output)? Maybe limit this to read
commands.

A way to let higher-level code avoid certain `file-exists-p' calls: add a
dynamic variable that tells TRAMP to skip certain commands if the result is
not available from a cache. Something similar to
`process-file-side-effects'. Calling code could then do sth. like this:

    (when (let ((tramp-may-skip-if-not-cached `((file-exists-p unknown
,file))))
            (file-exists-p file))  ; TRAMP will return t or nil if it knows
or 'unknown if not cached; for local files there is no effect
      ...)

Suggested semantics: list of (FUNCTION INSTANT-RESULT-IF-NOT-CACHED
ARGUMENT...). Any element of the list with unknown function name etc. would
be simply ignored.

Code that doesn't let-bind this variable will behave as before. Code that
cares can be optimized.

Paul

On Sat, 2 Jul 2022 at 17:58, Michael Albinus <michael.albinus@gmx.de> wrote:

> Paul Pogonyshev <pogonyshev@gmail.com> writes:
>
> Hi Paul,
>
> > 1) check if connection is alive (`echo are you awake');
> > 2) test if the file exists;
> > 3) creating a temporary file for the chunk to be inserted; I guess it
> > tries until it finds an unused filename, e.g. here it seems to be done
> > after `test -e /tmp/tramp.OD3cCu', which doesn't exist;
> > 4) 'touch' on the temporary file, presumably to create it;
> > 5) 'chmod' on the temporary, presumably so that other users cannot
> > read it;
> > 6) copying the requested chunk from the full file into the temporary
> > (using `dd');
> > 7) finding the real name of the temporary with `readlink';
> > 8) finding attributes of the temporary with `stat';
> > 9) gzipping the temporary for transmition from the remote to the local
> > machine;
> > 10) testing if the temporary is a directory (WTF?);
> > 11) removing the temporary.
> >
> > I guess it should be obvious that this is a bit too much for one
> > `insert-file-contents' call.
>
> In general, I agree. However, some of the commands are caused by
> primitive file operations, like file-exists-p. Tramp cannot know what
> will be the next call, and it doesn't have all the opportunities to
> optimize, compared with the overall picture you see in the eleven steps.
>
> > Suggested improvements:
> >
> > * TRAMP should issue just one `stat' command to find out most of the
> > things about a file: whether it exists, if it is a directory, its real
> > name when dereferencing links and whatever stats it is used to find
> > now; from `$ stat --help' this seems to be possible. In other words,
> > TRAMP shouldn't use simple commands like `test -e': any ping, even
> > nominal, will negate any gains from using a tad faster command.
> > Instead, if it needs to find anything about a file, it should ask the
> > remote about as many things as possible in one go: it is very likely
> > that the additional information will be needed soon and even if not,
> > this is basically free compared to ping anyway.
>
> Not all remote hosts carry a stat command, and not all existing stat's
> are GNU compatible. But yes, if possible, Tramp shall gather as much
> information in one run, and cache the results for further use.
>
> I will see what could be done. Will come back with a proposal next days
> (note that this will be for Emacs 29, ie git master).
>
> > * TRAMP code should prefer the approach "try do something and handle
> > resulting errors" where possible. For example, don't check if the file
> > exists, try to read it right away and handle failures properly. Code
> > like `(when (file-exists-p ...) do-something)' adds an unnecessary
> > command call and creates a racing condition anyway. Also, error-free
> > requests should be more frequent, so they should be the main
> > optimization goal. I'm not sure if it is applicable to TRAMP itself
> > and doesn't come from a higher level, though.
>
> Indeed, this is not Tramp's responsibility. Tramp is a stupid
> library. If there is a call for file-exists-p, it must return the
> answer. It doesn't know what will be the next request. So I'm rather
> pesimistic that Tramp can improve here.
>
> Best regards, Michael.
>

[-- Attachment #2: Type: text/html, Size: 5380 bytes --]