From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Paul Pogonyshev Newsgroups: gmane.emacs.bugs Subject: bug#56342: TRAMP (sh) issues way too many commands, thus being very slow over high-ping networks Date: Tue, 26 Jul 2022 10:00:40 +0200 Message-ID: References: <8735fjh5ge.fsf@gmx.de> <87v8sefl2f.fsf@gmx.de> <874jzyc9u9.fsf@gmx.de> <87sfnhazvy.fsf@gmx.de> <87k08sc02z.fsf@gmx.de> Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="000000000000fd113c05e4b0b222" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="26010"; mail-complaints-to="usenet@ciao.gmane.io" Cc: 56342@debbugs.gnu.org To: Michael Albinus Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Tue Jul 26 10:01:48 2022 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1oGFVk-0006ba-2f for geb-bug-gnu-emacs@m.gmane-mx.org; Tue, 26 Jul 2022 10:01:48 +0200 Original-Received: from localhost ([::1]:45186 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1oGFVi-0000Wu-VY for geb-bug-gnu-emacs@m.gmane-mx.org; Tue, 26 Jul 2022 04:01:47 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:44320) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oGFV1-0000TW-Gn for bug-gnu-emacs@gnu.org; Tue, 26 Jul 2022 04:01:05 -0400 Original-Received: from debbugs.gnu.org ([209.51.188.43]:34167) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1oGFV0-00028M-9P for bug-gnu-emacs@gnu.org; Tue, 26 Jul 2022 04:01:03 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1oGFV0-0007nw-2l for bug-gnu-emacs@gnu.org; Tue, 26 Jul 2022 04:01:02 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Paul Pogonyshev Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Tue, 26 Jul 2022 08:01:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 56342 X-GNU-PR-Package: emacs Original-Received: via spool by 56342-submit@debbugs.gnu.org id=B56342.165882246129986 (code B ref 56342); Tue, 26 Jul 2022 08:01:02 +0000 Original-Received: (at 56342) by debbugs.gnu.org; 26 Jul 2022 08:01:01 +0000 Original-Received: from localhost ([127.0.0.1]:52149 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1oGFUy-0007na-QU for submit@debbugs.gnu.org; Tue, 26 Jul 2022 04:01:01 -0400 Original-Received: from mail-yw1-f174.google.com ([209.85.128.174]:42818) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1oGFUw-0007nM-GD for 56342@debbugs.gnu.org; Tue, 26 Jul 2022 04:00:59 -0400 Original-Received: by mail-yw1-f174.google.com with SMTP id 00721157ae682-2ef5380669cso133936907b3.9 for <56342@debbugs.gnu.org>; Tue, 26 Jul 2022 01:00:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=xVieKpmRoq5vdeEhQ7eKL8ARrdBsLUXHYEtEFB7VEaI=; b=ZyZfdQl4RCtaOFEdGNIzTkXKKlz6C7ED8Z/mhdTjzOCnZVkxx9QamU+loiIEcup1Ra Sa9g9Qu3h2xumBGhjlSrOFcF5cpGZckuiYgG26FZ+vy5rTibfGPf0N3xOgk1sNGavltK 8TtKxCl4C8BBh1a5eYv7TFPV2XWTweqmcah03dUoD3jv2NzB7Y22Zh8n+in74tKsWRbF ucDzE2NCCRbJdMfk+OggXtj3AFH9uIp9gZv2Pooi2dISsbeNLypOTH+vivuHfcOarxGQ RmiH+LmVDYOGOmDZCFEoxDATUc3Vn/BXpIi+ToKCqb+kKNWZxumlKPYgL/ndP2pyl9Wz Cv7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=xVieKpmRoq5vdeEhQ7eKL8ARrdBsLUXHYEtEFB7VEaI=; b=CVvf0SjVp9VTMvL60MiTg/QBYvA3brKVaEsealtdv+XysoKZCVQlMh1S9k+FFDUP5G EdRGbWOPET2JtrCOCVJUB4Wo5zsX2OgPa0HKSwtMlwS7+5AbcqXqxto7kvQmw0XlwICy jT0WHE1x/faXyFX1Jl/AlSVtimwRLtzD6lZE4VjqSGQTiT72C6uFdOCsB4oXaB3RejCe VwPRDmmIoLDnMWoXwCSGFbrDneoHq4I+VtVvDTy4+TzCBmgfxyoRzkv4enJWh8qJaSuk r3FbKu5q0r8039cdNVVHxub40Hya2mwqjkv4F+xUfoa8CTV4gP+6MUBqGaM2no5CKXL9 PIIg== X-Gm-Message-State: AJIora+VORXQtmN+qrHzNhKMyjDeip0iqTM3y3ttdg2IntXIVY28k9YI YiiSoIsadKgcmU5vojq/xz5D878Jtkd4yPGMFg== X-Google-Smtp-Source: AGRyM1sBdFFx5CvD5NGbFppHeqCxt23d6ljLX2D7OFvwjyBosi07TKUGWWrcXbjcDXqpx+UVk/1H3C4Hv31Xioe5qtg= X-Received: by 2002:a0d:df04:0:b0:31e:573a:37f4 with SMTP id i4-20020a0ddf04000000b0031e573a37f4mr13649498ywe.76.1658822451858; Tue, 26 Jul 2022 01:00:51 -0700 (PDT) In-Reply-To: <87k08sc02z.fsf@gmx.de> X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.io gmane.emacs.bugs:237931 Archived-At: --000000000000fd113c05e4b0b222 Content-Type: text/plain; charset="UTF-8" Hello, any news on this? Paul On Mon, 4 Jul 2022 at 18:30, Michael Albinus wrote: > Paul Pogonyshev writes: > > Hi Paul, > > >> It returns "'/tmp/c' -> '/tmp/b'". However, we need "/tmp/a". So we must > >> still use "readlink --canonicalize". > > > > According to a quick search, it is possible to merge output of several > > shell commands together. This seems to work even with dumb `sh', it's > > not a Bash extension: > > > > $ sh -c '{ stat xxx && readlink xxx; }' > > > > I guess TRAMP could just sth. similar, as I understand it doesn't have > > to be strictly one executable call, just one command given to the > > remote shell. > > I'll check, perhaps it could be used in this special case. > > But Tramp uses already combination of shell commands, and it uses even > private shell functions when applicable. Don't expect too much in > general. > > >> Tramps communication with the remote host is like a REPL engine. It > >> sends shell commands to the remote hosts, reads the result, and waits > >> for the shell prompt. If it doesn't wait for the final shell prompt, it > >> is likely that the result or the shell prompt will be seen when reading > >> the output of the next command. This confuses. So no, I don't see a > >> chance to implement this kind of "asynchronity". > > > > I see parameter `nooutput' to `tramp-send-command'. Couldn't that be > > used? > > No. This argument tells tramp-send-command not to call the final > accept-process-output. But this is only because on the caller side, > accept-process-output will be called with other arguments but the > default ones. > > > Even if not, I could imagine sth. like this: > > > > (defvar pending-commands nil) > > (defvar reading-output nil) > > > > (defun send-command (x output-inessential) > > (if output-inessential > > (setf pending-commands (nconc pending-commands (list x))) > > (while reading-output ; make sure the connection is free for > > the next essential command > > (read-next-output-chunk) > > (when (and (not reading-output) pending-commands) > > (do-send-command (pop pending-commands)))) > > (do-send-command x) > > (read-output-now))) > > > > (defun do-send-command (x) > > (really-do-send-it x) > > (setf reading-output t)) > > > > (defun read-output-now () > > (while reading-output > > (read-next-output-chunk)) > > (extract-received-output-from-process-buffer)) > > > > (defun emacs-idling () ; hooked up using `run-with-idle-timer' or > > something like that > > (cond (reading-output > > (read-next-output-chunk)) > > (pending-commands > > (do-send-command (pop pending-commands))))) > > > > (defun read-next-output-chunk () > > (when reading-output > > (do-read-output-chunk) ; this should be non-blocking > > (when (end-of-command-output) > > (setf reading-output nil)))) > > Something like this could be done, yes. Perhaps not with an (idle) > timer, because they are known to interrupt Tramp's REPL engine at any > point, with (sometimes) desastrous results. In general, one shall try to > avoid Tramp actions in timers, process sentinels and process filters. > > So we could use this for deleting temporary files and alike, but there > won't be too many performance boosters I fear. > > I will play with this idea, perhaps it helps. But for sure it will be an > opt-in feature only. > > > Paul > > Best regards, Michael. > --000000000000fd113c05e4b0b222 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hello,

any news on this?

=
Paul

On Mon, 4 Jul 2022 at 18:30, Michael Albinus <michael.albinus@gmx.de> wrote:<= br>
Paul Pogonyshev = <pogonyshev@gm= ail.com> writes:

Hi Paul,

>> It returns "'/tmp/c' -> '/tmp/b'". Ho= wever, we need "/tmp/a". So we must
>> still use "readlink --canonicalize".
>
> According to a quick search, it is possible to merge output of several=
> shell commands together. This seems to work even with dumb `sh', i= t's
> not a Bash extension:
>
>=C2=A0 =C2=A0 =C2=A0$ sh -c '{ stat xxx && readlink xxx; }&= #39;
>
> I guess TRAMP could just sth. similar, as I understand it doesn't = have
> to be strictly one executable call, just one command given to the
> remote shell.

I'll check, perhaps it could be used in this special case.

But Tramp uses already combination of shell commands, and it uses even
private shell functions when applicable. Don't expect too much in
general.

>> Tramps communication with the remote host is like a REPL engine. I= t
>> sends shell commands to the remote hosts, reads the result, and wa= its
>> for the shell prompt. If it doesn't wait for the final shell p= rompt, it
>> is likely that the result or the shell prompt will be seen when re= ading
>> the output of the next command. This confuses. So no, I don't = see a
>> chance to implement this kind of "asynchronity".
>
> I see parameter `nooutput' to `tramp-send-command'. Couldn'= ;t that be
> used?

No. This argument tells tramp-send-command not to call the final
accept-process-output. But this is only because on the caller side,
accept-process-output will be called with other arguments but the
default ones.

> Even if not, I could imagine sth. like this:
>
>=C2=A0 =C2=A0 =C2=A0(defvar pending-commands nil)
>=C2=A0 =C2=A0 =C2=A0(defvar reading-output nil)
>
>=C2=A0 =C2=A0 =C2=A0(defun send-command (x output-inessential)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0(if output-inessential
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(setf pending-commands (nconc = pending-commands (list x)))
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(while reading-output=C2=A0 ; make su= re the connection is free for
> the next essential command
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(read-next-output-chunk)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(when (and (not reading-output= ) pending-commands)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(do-send-command (pop p= ending-commands))))
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(do-send-command x)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(read-output-now)))
>
>=C2=A0 =C2=A0 =C2=A0(defun do-send-command (x)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0(really-do-send-it x)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0(setf reading-output t))
>
>=C2=A0 =C2=A0 =C2=A0(defun read-output-now ()
>=C2=A0 =C2=A0 =C2=A0 =C2=A0(while reading-output
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(read-next-output-chunk))
>=C2=A0 =C2=A0 =C2=A0 =C2=A0(extract-received-output-from-process-buffer= ))
>
>=C2=A0 =C2=A0 =C2=A0(defun emacs-idling ()=C2=A0 ; hooked up using `run= -with-idle-timer' or
> something like that
>=C2=A0 =C2=A0 =C2=A0 =C2=A0(cond (reading-output
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (read-next-output-chun= k))
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(pending-commands
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (do-send-command (pop = pending-commands)))))
>
>=C2=A0 =C2=A0 =C2=A0(defun read-next-output-chunk ()
>=C2=A0 =C2=A0 =C2=A0 =C2=A0(when reading-output
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(do-read-output-chunk)=C2=A0 ; this s= hould be non-blocking
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(when (end-of-command-output)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(setf reading-output nil))))
Something like this could be done, yes. Perhaps not with an (idle)
timer, because they are known to interrupt Tramp's REPL engine at any point, with (sometimes) desastrous results. In general, one shall try to avoid Tramp actions in timers, process sentinels and process filters.

So we could use this for deleting temporary files and alike, but there
won't be too many performance boosters I fear.

I will play with this idea, perhaps it helps. But for sure it will be an opt-in feature only.

> Paul

Best regards, Michael.
--000000000000fd113c05e4b0b222--