From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Paul Pogonyshev Newsgroups: gmane.emacs.bugs Subject: bug#56342: TRAMP (sh) issues way too many commands, thus being very slow over high-ping networks Date: Tue, 26 Jul 2022 18:17:01 +0200 Message-ID: References: <8735fjh5ge.fsf@gmx.de> <87v8sefl2f.fsf@gmx.de> <874jzyc9u9.fsf@gmx.de> <87sfnhazvy.fsf@gmx.de> <87k08sc02z.fsf@gmx.de> <87r128gdob.fsf@gmx.de> Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="0000000000001084de05e4b7a234" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="16972"; mail-complaints-to="usenet@ciao.gmane.io" Cc: 56342@debbugs.gnu.org To: Michael Albinus Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Tue Jul 26 18:54:28 2022 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1oGNpE-0004C9-Np for geb-bug-gnu-emacs@m.gmane-mx.org; Tue, 26 Jul 2022 18:54:28 +0200 Original-Received: from localhost ([::1]:37500 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1oGNpD-0007C3-Pu for geb-bug-gnu-emacs@m.gmane-mx.org; Tue, 26 Jul 2022 12:54:27 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:36892) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oGNG0-0002He-Sc for bug-gnu-emacs@gnu.org; Tue, 26 Jul 2022 12:18:05 -0400 Original-Received: from debbugs.gnu.org ([209.51.188.43]:36283) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1oGNFy-00077Q-CL for bug-gnu-emacs@gnu.org; Tue, 26 Jul 2022 12:18:03 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1oGNFy-0006wK-7v for bug-gnu-emacs@gnu.org; Tue, 26 Jul 2022 12:18:02 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Paul Pogonyshev Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Tue, 26 Jul 2022 16:18:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 56342 X-GNU-PR-Package: emacs Original-Received: via spool by 56342-submit@debbugs.gnu.org id=B56342.165885224126612 (code B ref 56342); Tue, 26 Jul 2022 16:18:02 +0000 Original-Received: (at 56342) by debbugs.gnu.org; 26 Jul 2022 16:17:21 +0000 Original-Received: from localhost ([127.0.0.1]:54265 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1oGNFI-0006vA-Eq for submit@debbugs.gnu.org; Tue, 26 Jul 2022 12:17:21 -0400 Original-Received: from mail-yw1-f173.google.com ([209.85.128.173]:37388) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1oGNFG-0006uw-KC for 56342@debbugs.gnu.org; Tue, 26 Jul 2022 12:17:19 -0400 Original-Received: by mail-yw1-f173.google.com with SMTP id 00721157ae682-31e623a4ff4so148213827b3.4 for <56342@debbugs.gnu.org>; Tue, 26 Jul 2022 09:17:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=/r5CgC8aZteyQrDEsi6qi4Y/pEmHhZCYKcvPhHtGS8A=; b=VQS53uKm0UQkaDMASdiaaEYYFG2dmOKuGQRLWxPAs6JU4ZNgQnw+rHpMG7lX/k/cY/ qNFl80PLbPqouhYXDT5vXtL+LMbyi0rN1v7upb3PuBVHTJtg9tBRfTqtdbjQJOJLG+M/ 0FxTrKEMbdQkzZMQE9ITLMe7IQyQmc8VBAoyweQ17gZuCc5FRgFU56zLPo4bLYt8Tmoc 9qt/4hNnlnkwE/Kl+YhM4OUqD54YUo7Vf16VXM4svxxdx3qyVWNaF3aTBjdyyJrB14Z4 l0kzPcZVEJsVCMUJQ8VA454uEH2r4bPojiqM1CqiuWstInN+L0wja9t0rtvwAHKqup5u BvEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=/r5CgC8aZteyQrDEsi6qi4Y/pEmHhZCYKcvPhHtGS8A=; b=O9L6eA66w/6w4aYJp/94HTmEyaq/YDF5ScMMCxOnnkecdsj5fbtSyl5evSeirGz309 k+lJOa9uhG7By+1TnV+72R2a7D/G97MJ5WdMqkxBL+N3FmKk565CB/3cJ6APi7akHuHE 7s/C01wrbbyfJy91M68RBxW4PpFgLpP53OnAHxJo4lbMqmboPKcOA6YEkrxRr3ZQ5tdk cbz4nAvNAHmzSLPhaJqnxJrPz6VJIP5f5Mc3aaZpvWSlh7MrU79dzb6jPEdSPujWOsaQ 0OrVmua/SKDs6olwq9DNA/ChBsfVqrO9B9w804zDY72wyhqxGxWTD5VqPIoHQavvREOy rx/Q== X-Gm-Message-State: AJIora9iMqAjVYDw1CaDfRECXl7a6djBTk87ZIJPqoMx0QOU1UU7rfjI 6F+SzP/OUo78fKj7AOW3NLujR7Tys+TC8rmXug== X-Google-Smtp-Source: AGRyM1uw6MZ/5hL6b1X7Kmdu7tmY5wvROD1LSdIR9qXHvmnZbbPrGg6sKJT7kPlKz5WSeVDCqDHqcuwk6Lhstypu5LQ= X-Received: by 2002:a81:1c0a:0:b0:31e:6ab0:7f73 with SMTP id c10-20020a811c0a000000b0031e6ab07f73mr15336238ywc.353.1658852232691; Tue, 26 Jul 2022 09:17:12 -0700 (PDT) In-Reply-To: <87r128gdob.fsf@gmx.de> X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.io gmane.emacs.bugs:237978 Archived-At: --0000000000001084de05e4b7a234 Content-Type: text/plain; charset="UTF-8" > I've worked last weeks almost full time on this. Thank you! > Combining stat and readlink into one remote command have even slowed > down the machinery. Likely, because it isn't needed always, but it > always consumes more time to compute both on the remote side, and to > read all the results. Well, that's hard to believe. I mean, it is understandable that `readlink' (whatever Elisp call results in that shell command) is not often needed, but how long can that take on a modern computer, 1 ms? It might be that we evaluate performance changes differently. I care more about worst cases (since they affect me directly, yeah) and not so much if Tramp becomes 1 ms slower "per user-visible operation" on average. > Perhaps you give it a try to see, whether your use cases behave better. I tested it a bit now. Initially mentioned usecase (`insert-file-contents', but only a file part, not the whole file) appears to be considerably faster after the changes (hard to say, feels like 30-50% faster). The connection log says that number of commands is 10 now, with the originally mentioned list heavily changed. However, there must still be space for improvement, e.g. `M-x occur tramp-send RET' suggests there are outright duplicated commands now: 18:03:41.617738 tramp-send-command (6) # \readlink --canonicalize-missing /tmp/tramp.H3AZYb 2>/dev/null; echo tramp_exit_status $? 18:03:41.650294 tramp-send-command (6) # tramp_stat_file_attributes /tmp/tramp.H3AZYb 2>/dev/null; echo tramp_exit_status $? 18:03:41.699627 tramp-send-command (6) # base64 -d -i >/tmp/tramp.H3AZYb <<'9190079abe64738d52b0f040fd94461c' 2>/dev/null; echo tramp_exit_status $? 18:03:41.731669 tramp-send-command (6) # chown 1000:1001 /tmp/tramp.H3AZYb 18:03:41.760145 tramp-send-command (6) # dd bs=1 skip=8166969 if=[...] of=/tmp/tramp.H3AZYb 18:03:41.843785 tramp-send-command (6) # \readlink --canonicalize-missing /tmp/tramp.H3AZYb 2>/dev/null; echo tramp_exit_status $? 18:03:41.872072 tramp-send-command (6) # tramp_stat_file_attributes /tmp/tramp.H3AZYb 2>/dev/null; echo tramp_exit_status $? 18:03:41.902388 tramp-send-command (6) # (env GZIP= gzip /dev/null; echo tramp_exit_status $? 18:03:41.952283 tramp-send-command (6) # rm -f /tmp/tramp.H3AZYb 2>/dev/null; echo tramp_exit_status $? On the other hand, refreshing a remote-tracking Magit (Emacs Git interface) buffer doesn't feel any faster than before. Logs also suggest that number of commands that tramp send stays as before (43). Some commands are certainly excessive, e.g. it repeats `test -d [...] 2>/dev/null; echo tramp_exit_status $?' 4 times for the same directory (intermixed with other commands), suggesting that Tramp doesn't cache the result. Three times it issues `test -r ...' for the same directory, the source checkout root. Most other commands cannot be generically skipped by Tramp though, this would rather depend on Magit. Paul On Tue, 26 Jul 2022 at 16:18, Michael Albinus wrote: > Paul Pogonyshev writes: > > > Hello, > > Hi Paul, > > > any news on this? > > I've worked last weeks almost full time on this. With limited success. > > Combining stat and readlink into one remote command have even slowed > down the machinery. Likely, because it isn't needed always, but it > always consumes more time to compute both on the remote side, and to > read all the results. > > Moving the check of the existence of a file to the end, and let the > commands fail in case the file doesn't exist, isn't possible. Often, > Tramp needs to know in advance the size of a given file, for example to > decide whether to encode/decode the file or to invoke scp, or to know > whether the file shall be compressed prior transfer. Checking the file > size is much the same as checking the file existence. > > Avoiding the "are you awake" command is still on my todo list, but it > isn't trivial. There is a complex machinery to be invoked in case the > command fails, in order to reestablish the connection in the > background. It still might be possible to change this, but it isn't > trivial. > > Bu there are also some results. With commit > 9ed5c39aad09571314097be91cb28e7504614421 I've refactored major Tramp > parts in order to have a better chance to apply performance > changes. This includes already some changes to combine several commands > (avoiding superfluous roundtrips) and alike. There's also commit > dfa16cadc18930fad76fa6113750eaa27d367e72 fixing a regression which was > introduced with the former patch. > > In my test environment, performance improvement of these changes is not > overwhelming. But I sit in a LAN, with good connections. Perhaps you > give it a try to see, whether your use cases behave better. > > I'll continue to work on this, but don't expect wonders. > > > Paul > > Best regards, Michael. > --0000000000001084de05e4b7a234 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
> I've worked last weeks almost full time on this.<= br>

Thank you!

> Combining s= tat and readlink into one remote command have even slowed
>=C2=A0down= the machinery. Likely, because it isn't needed always, but it
>= =C2=A0always consumes more time to compute both on the remote side, and to<= br>>=C2=A0read all the results.

Well, that&= #39;s hard to believe. I mean, it is understandable that `readlink' (wh= atever Elisp call results in that shell command) is not often needed, but h= ow long can that take on a modern computer, 1 ms? It might be that we evalu= ate performance changes differently. I care more about worst cases (since t= hey affect me directly, yeah) and not so much if Tramp becomes 1 ms slower = "per user-visible operation" on average.

> Perhaps you
give it a try to see, whether your use cases behave be= tter.

I tested it a bit now. Initially mention= ed usecase (`insert-file-contents', but only a file part, not the whole= file) appears to be considerably faster after the changes (hard to say, fe= els like 30-50% faster). The connection log says that number of commands is= 10 now, with the originally mentioned list heavily changed. However, there= must still be space for improvement, e.g. `M-x occur tramp-send RET' s= uggests there are outright duplicated commands now:

=C2=A0 =C2=A0 18:03:41.617738 tramp-send-command (6) # \readlink --canoni= calize-missing /tmp/tramp.H3AZYb 2>/dev/null; echo tramp_exit_status $?<= br>=C2=A0 =C2=A0 18:03:41.650294 tramp-send-command (6) # tramp_stat_file_a= ttributes /tmp/tramp.H3AZYb 2>/dev/null; echo tramp_exit_status $?
= =C2=A0 =C2=A0 18:03:41.699627 tramp-send-command (6) # base64 -d -i >/tm= p/tramp.H3AZYb <<'9190079abe64738d52b0f040fd94461c' 2>/dev= /null; echo tramp_exit_status $?
=C2=A0 =C2=A0 18:03:41.731669 tramp-sen= d-command (6) # chown 1000:1001 /tmp/tramp.H3AZYb
=C2=A0 =C2=A0 18:03:41= .760145 tramp-send-command (6) # dd bs=3D1 skip=3D8166969 if=3D[...] of=3D/= tmp/tramp.H3AZYb
=C2=A0 =C2=A0 18:03:41.843785 tramp-send-command (6) # = \readlink --canonicalize-missing /tmp/tramp.H3AZYb 2>/dev/null; echo tra= mp_exit_status $?
=C2=A0 =C2=A0 18:03:41.872072 tramp-send-command (6) #= tramp_stat_file_attributes /tmp/tramp.H3AZYb 2>/dev/null; echo tramp_ex= it_status $?
=C2=A0 =C2=A0 18:03:41.902388 tramp-send-command (6) # (env= GZIP=3D gzip </tmp/tramp.H3AZYb | base64) 2>/dev/null; echo tramp_ex= it_status $?
=C2=A0 =C2=A0 18:03:41.952283 tramp-send-command (6) # rm -= f /tmp/tramp.H3AZYb 2>/dev/null; echo tramp_exit_status $?

On the other hand, refreshing a remote-tracking Magit (Emac= s Git interface)=C2=A0buffer=C2=A0doesn't feel any faster than before. = Logs also suggest that number of commands that tramp send stays as before (= 43). Some commands are certainly excessive, e.g. it repeats `test -d [...] = 2>/dev/null; echo tramp_exit_status $?' 4 times for the same directo= ry (intermixed with other commands), suggesting that Tramp doesn't cach= e the result. Three times it issues `test -r ...' for the same director= y, the source checkout root. Most other commands cannot be generically skip= ped by Tramp though, this would rather depend on Magit.

Paul

On Tue, 26 Jul 2022 at 16:18, Michael Albinus <michael.albinus@gmx.de> wrote:
Paul Pogonyshev &l= t;pogonyshev@gmai= l.com> writes:

> Hello,

Hi Paul,

> any news on this?

I've worked last weeks almost full time on this. With limited success.<= br>
Combining stat and readlink into one remote command have even slowed
down the machinery. Likely, because it isn't needed always, but it
always consumes more time to compute both on the remote side, and to
read all the results.

Moving the check of the existence of a file to the end, and let the
commands fail in case the file doesn't exist, isn't possible. Often= ,
Tramp needs to know in advance the size of a given file, for example to
decide whether to encode/decode the file or to invoke scp, or to know
whether the file shall be compressed prior transfer. Checking the file
size is much the same as checking the file existence.

Avoiding the "are you awake" command is still on my todo list, bu= t it
isn't trivial. There is a complex machinery to be invoked in case the command fails, in order to reestablish the connection in the
background. It still might be possible to change this, but it isn't
trivial.

Bu there are also some results. With commit
9ed5c39aad09571314097be91cb28e7504614421 I've refactored major Tramp parts in order to have a better chance to apply performance
changes. This includes already some changes to combine several commands
(avoiding superfluous roundtrips) and alike. There's also commit
dfa16cadc18930fad76fa6113750eaa27d367e72 fixing a regression which was
introduced with the former patch.

In my test environment, performance improvement of these changes is not
overwhelming. But I sit in a LAN, with good connections. Perhaps you
give it a try to see, whether your use cases behave better.

I'll continue to work on this, but don't expect wonders.

> Paul

Best regards, Michael.
--0000000000001084de05e4b7a234--