Re: 22.1.50; insert-file-contents is slow under tramp

[Michael Albinus <michael.albinus@gmx.de>]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Yes. I've installed a patch in the trunk, which makes a fast track for
>> "remote" files being on localhost.
>
> How does that work?  I mean when accessing files on the localhost with
> Tramp, the most likely situation I can think of is the `su' method so how
> what does your "fast track" do to transfer the file faster?  Does it copy it
> to /tmp at the "remote" end, then make it world-readable, then copy it on
> the "local" end, then remove it at the remote end?

Similar approach. It calls `tramp-do-copy-or-rename-file-directly',
which copies the "remote" file to /tmp, and assigns proper
permissions. Then `insert-file-contents' for the _local_ temporary file
is applied, which shows the file in the buffer, and the temporary file
is deleted.

>> For real remote hosts, I recommend to use scp when accessing huge files.
>
> Reminds me: where is the bottleneck in the inline method?  I mean
> fundamentally doing a base64 encoding/decoding shouldn't cost that much
> (negligible CPU-wise and a mere 33% in network transfer).  Maybe we could
> attack this part to improve performance?

I haven't investigated too much. mimencode and base64-decode are pretty
fast, even on my stone-age machine, with an 8MB sized file:

| (with-current-buffer (get-buffer-create "foo")
|   (elp-instrument-function 'call-process)
|   (elp-instrument-function 'base64-decode-region)
|   (erase-buffer)
|
|   (call-process "mimencode" nil t nil "/var/cache/apt/pkgcache.bin")
|   (base64-decode-region (point-min) (point-max))
|   (elp-results))
|
| Function Name         Call Count  Elapsed Time  Average Time
| ====================  ==========  ============  ============
| base64-decode-region  1           7.732203      7.732203
| call-process          1           1.917887      1.917887

But with the installed "fast track", it is remarkable faster than even
these two basic functions:

| (with-current-buffer (get-buffer-create "foo")
|   (elp-instrument-function 'insert-file-contents)
|   (erase-buffer)
|   (setq tramp-verbose 0)
|
|   ;; This opens the connection, including password handover etc
|   (file-attributes "/sudo::")
|
|   (insert-file-contents "/sudo::/var/cache/apt/pkgcache.bin")
|   (elp-results))
|
| Function Name         Call Count  Elapsed Time  Average Time
| ====================  ==========  ============  ============
| insert-file-contents  2           1.64688       0.82344

Without the fast track, the access time is inacceptable large. I suspect
that `tramp-wait-for-output' is the bottleneck, because it must check
for the resulting shell prompt, handle echoing shells, etc. I will
investigate next days whether there can be an improvement.

>         Stefan

Best regards, Michael.