From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Paul Pogonyshev Newsgroups: gmane.emacs.bugs Subject: bug#56342: TRAMP (sh) issues way too many commands, thus being very slow over high-ping networks Date: Mon, 4 Jul 2022 16:42:26 +0200 Message-ID: References: <8735fjh5ge.fsf@gmx.de> <87v8sefl2f.fsf@gmx.de> <874jzyc9u9.fsf@gmx.de> <87sfnhazvy.fsf@gmx.de> Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="00000000000056054c05e2fbbf5c" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="17551"; mail-complaints-to="usenet@ciao.gmane.io" Cc: 56342@debbugs.gnu.org To: Michael Albinus Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Mon Jul 04 16:51:34 2022 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1o8NQE-0004OW-At for geb-bug-gnu-emacs@m.gmane-mx.org; Mon, 04 Jul 2022 16:51:34 +0200 Original-Received: from localhost ([::1]:40230 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1o8NQC-0001Xd-Mt for geb-bug-gnu-emacs@m.gmane-mx.org; Mon, 04 Jul 2022 10:51:32 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:58162) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1o8NHy-0006nb-2v for bug-gnu-emacs@gnu.org; Mon, 04 Jul 2022 10:43:02 -0400 Original-Received: from debbugs.gnu.org ([209.51.188.43]:54488) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1o8NHx-000192-Qq for bug-gnu-emacs@gnu.org; Mon, 04 Jul 2022 10:43:01 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1o8NHx-00061A-O8 for bug-gnu-emacs@gnu.org; Mon, 04 Jul 2022 10:43:01 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Paul Pogonyshev Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Mon, 04 Jul 2022 14:43:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 56342 X-GNU-PR-Package: emacs Original-Received: via spool by 56342-submit@debbugs.gnu.org id=B56342.165694576723101 (code B ref 56342); Mon, 04 Jul 2022 14:43:01 +0000 Original-Received: (at 56342) by debbugs.gnu.org; 4 Jul 2022 14:42:47 +0000 Original-Received: from localhost ([127.0.0.1]:48379 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1o8NHj-00060U-2V for submit@debbugs.gnu.org; Mon, 04 Jul 2022 10:42:47 -0400 Original-Received: from mail-yw1-f177.google.com ([209.85.128.177]:33289) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1o8NHg-000605-1A for 56342@debbugs.gnu.org; Mon, 04 Jul 2022 10:42:45 -0400 Original-Received: by mail-yw1-f177.google.com with SMTP id 00721157ae682-31c89111f23so33827777b3.0 for <56342@debbugs.gnu.org>; Mon, 04 Jul 2022 07:42:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=jr/B1g9LGAvqvTYCa0EDnel71tzPyxGDbO2Mt+h1vXg=; b=aVLdYbFOrw/Mvt6pMTOFwKu2t/KdMW9zzbpfuhz50F8S/GRnAz46IkZEpKDQs+NMSt tW7UzegHDhwOJowEXPLlw1jvVQ9YgWpQY8V7vLtkpal7BqoBP4AHq+MMVn2p4h8bWMGf coTferDfS4ZcBk80jza+PQjGGu0Ialwl5WX1fdDJyRzd99oBRy8xNoUZB59ZQdCbEK2U 617kv/iIh68LNH7GBZlmlt0jCcOton0Xcau8cbIq9/a0kU3qUZS3dwYf0u/Y/HLddwlK rDI87HggTP3XFYwRTz2FRhL3YQiZlSQ0vB1fW5Y2MjmsQcGCNiGx2BWS8NWxytdLv1Uq wdqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=jr/B1g9LGAvqvTYCa0EDnel71tzPyxGDbO2Mt+h1vXg=; b=UEK8ov/YQH35vhgO9AHdHfTCSEw/WdwqmWZbGeZ5bDcfQ7DTwFw3NLmHEfg+petPwc +lM5mAC2u7yI3FffNlcs+zz0y5bAnoqUMxSaQJUwiS/hHo3PX+jwyZ+KugJIJ5gB7ELG bj0ofpMWlc60Ouk1K5k9AoHHRjwjQPrRjRNY+BiJrILwLfdIQI/Ohts/PimBCCWSGOCE QECb/woHIacUaxtIhup20V+38uSv0Xop0rfiJUvGYWgHuMacCICDxNMSTcGa4NMfyIwN +qVF6fX66hkcvFMTpT38R5Mxwl6SmjtL9QMENmvH09nmJodQztxZ6wHVsBnfQkHS6qse LetQ== X-Gm-Message-State: AJIora+QLFWRly+uz55ZmlpClm+++iXwvwJ6H6JyG0ieLe3qGoU/HMM6 kLqJcqQk7tnFYSg472iuXRjWE2wdKbeptHCEKC97jeSxmA== X-Google-Smtp-Source: AGRyM1sLDVjumx9IIZh7xBV0DKu+JO5nEBX5w6SveNf1SekWHkYzHsnwA4+yMFFZ7rg+bwdgLtCD4YD+ikYyK0Z7580= X-Received: by 2002:a81:106:0:b0:2d0:e682:8a7a with SMTP id 6-20020a810106000000b002d0e6828a7amr34241796ywb.257.1656945758306; Mon, 04 Jul 2022 07:42:38 -0700 (PDT) In-Reply-To: <87sfnhazvy.fsf@gmx.de> X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.io gmane.emacs.bugs:236061 Archived-At: --00000000000056054c05e2fbbf5c Content-Type: text/plain; charset="UTF-8" > It returns "'/tmp/c' -> '/tmp/b'". However, we need "/tmp/a". So we must > still use "readlink --canonicalize". According to a quick search, it is possible to merge output of several shell commands together. This seems to work even with dumb `sh', it's not a Bash extension: $ sh -c '{ stat xxx && readlink xxx; }' I guess TRAMP could just sth. similar, as I understand it doesn't have to be strictly one executable call, just one command given to the remote shell. > Tramps communication with the remote host is like a REPL engine. It > sends shell commands to the remote hosts, reads the result, and waits > for the shell prompt. If it doesn't wait for the final shell prompt, it > is likely that the result or the shell prompt will be seen when reading > the output of the next command. This confuses. So no, I don't see a > chance to implement this kind of "asynchronity". I see parameter `nooutput' to `tramp-send-command'. Couldn't that be used? Even if not, I could imagine sth. like this: (defvar pending-commands nil) (defvar reading-output nil) (defun send-command (x output-inessential) (if output-inessential (setf pending-commands (nconc pending-commands (list x))) (while reading-output ; make sure the connection is free for the next essential command (read-next-output-chunk) (when (and (not reading-output) pending-commands) (do-send-command (pop pending-commands)))) (do-send-command x) (read-output-now))) (defun do-send-command (x) (really-do-send-it x) (setf reading-output t)) (defun read-output-now () (while reading-output (read-next-output-chunk)) (extract-received-output-from-process-buffer)) (defun emacs-idling () ; hooked up using `run-with-idle-timer' or something like that (cond (reading-output (read-next-output-chunk)) (pending-commands (do-send-command (pop pending-commands))))) (defun read-next-output-chunk () (when reading-output (do-read-output-chunk) ; this should be non-blocking (when (end-of-command-output) (setf reading-output nil)))) Paul On Mon, 4 Jul 2022 at 13:19, Michael Albinus wrote: > Paul Pogonyshev writes: > > Hi Paul, > > >> Doing it asynchronously would require a second connection to the remote > >> host. Performance would rather degrade. > > > > Maybe not really asynchronously, just let it return early, not waiting > > for the result? I'm not sure how TRAMP receives responses, but can it > > just keep executing commands sequentially, as now, but give control > > back to the caller in case of commands where the result doesn't really > > matter (cleanup, e.g. deleting a temporary file). Of course, this > > means that if an "important" command is issued right away, it has to > > wait for the response to the previous inessential command. But when > > such an inessential command is the last in a batch, this waiting would > > be effectively merged with Emacs' idling in the normal UI command > > loop, making things more responsive for the user. > > Tramps communication with the remote host is like a REPL engine. It > sends shell commands to the remote hosts, reads the result, and waits > for the shell prompt. If it doesn't wait for the final shell prompt, it > is likely that the result or the shell prompt will be seen when reading > the output of the next command. This confuses. So no, I don't see a > chance to implement this kind of "asynchronity". > > > Paul > > Best regards, Michael. > --00000000000056054c05e2fbbf5c Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
> It returns "'/tmp/c' -> '/tmp/b&#= 39;". However, we need "/tmp/a". So we must
> still us= e "readlink --canonicalize".

According to = a quick search, it is possible to merge output of several shell commands to= gether. This seems to work even with dumb `sh', it's not a Bash ext= ension:

=C2=A0 =C2=A0 $ sh -c '{ stat xxx &= ;& readlink xxx; }'

I guess TRAMP coul= d just sth. similar, as I understand it doesn't have to be strictly one= executable call, just one command given to the remote shell.
> Tramps communication with the remote host is like a REPL e= ngine. It
>=C2=A0sends shell commands to the remote hosts, reads the = result, and waits
>=C2=A0for the shell prompt. If it doesn't wait= for the final shell prompt, it
>=C2=A0is likely that the result or t= he shell prompt will be seen when reading
>=C2=A0the output of the ne= xt command. This confuses. So no, I don't see a
>=C2=A0chance to = implement this kind of "asynchronity".

I see parameter `nooutput' to `tramp-send-command'. Couldn't= that be used?

Even if not, I could imagine sth. l= ike this:

=C2=A0 =C2=A0 (defvar pending-commands n= il)
=C2=A0 =C2=A0 (defvar reading-output nil)

=C2=A0 =C2=A0 (defu= n send-command (x output-inessential)
=C2=A0 =C2=A0 =C2=A0 (if output-in= essential
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (setf pending-commands (nco= nc pending-commands (list x)))
=C2=A0 =C2=A0 =C2=A0 =C2=A0 (while readin= g-output =C2=A0; make sure the connection is free for the next essential co= mmand
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (read-next-output-chunk)
=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (when (and (not reading-output) pending-com= mands)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (do-send-command (pop p= ending-commands))))
=C2=A0 =C2=A0 =C2=A0 =C2=A0 (do-send-command x)
= =C2=A0 =C2=A0 =C2=A0 =C2=A0 (read-output-now)))

=C2=A0 =C2=A0 (defun= do-send-command (x)
=C2=A0 =C2=A0 =C2=A0 (really-do-send-it x)
=C2= =A0 =C2=A0 =C2=A0 (setf reading-output t))

=C2=A0 =C2=A0 (defun read= -output-now ()
=C2=A0 =C2=A0 =C2=A0 (while reading-output
=C2=A0 =C2= =A0 =C2=A0 =C2=A0 (read-next-output-chunk))
=C2=A0 =C2=A0 =C2=A0 (extrac= t-received-output-from-process-buffer))

=C2=A0 =C2=A0 (defun emacs-i= dling () =C2=A0; hooked up using `run-with-idle-timer' or something lik= e that
=C2=A0 =C2=A0 =C2=A0 (cond (reading-output
=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(read-next-output-chunk))
=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 (pending-commands
=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0(do-send-command (pop pending-commands)))))

= =C2=A0 =C2=A0 (defun read-next-output-chunk ()
=C2=A0 =C2=A0 =C2=A0 (whe= n reading-output
=C2=A0 =C2=A0 =C2=A0 =C2=A0 (do-read-output-chunk) =C2= =A0; this should be non-blocking
=C2=A0 =C2=A0 =C2=A0 =C2=A0 (when (end-= of-command-output)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (setf reading-outp= ut nil))))

Paul

On Mon, 4 Jul 2022 at 13:19, = Michael Albinus <michael.albinus@gmx.de> wrote:
Paul Pogonyshev <pogonyshev@gmail.com> writes:

Hi Paul,

>> Doing it asynchronously would require a second connection to the r= emote
>> host. Performance would rather degrade.
>
> Maybe not really asynchronously, just let it return early, not waiting=
> for the result? I'm not sure how TRAMP receives responses, but can= it
> just keep executing commands sequentially, as now, but give control > back to the caller in case of commands where the result doesn't re= ally
> matter (cleanup, e.g. deleting a temporary file). Of course, this
> means that if an "important" command is issued right away, i= t has to
> wait for the response to the previous inessential command. But when > such an inessential command is the last in a batch, this waiting would=
> be effectively merged with Emacs' idling in the normal UI command<= br> > loop, making things more responsive for the user.

Tramps communication with the remote host is like a REPL engine. It
sends shell commands to the remote hosts, reads the result, and waits
for the shell prompt. If it doesn't wait for the final shell prompt, it=
is likely that the result or the shell prompt will be seen when reading
the output of the next command. This confuses. So no, I don't see a
chance to implement this kind of "asynchronity".

> Paul

Best regards, Michael.
--00000000000056054c05e2fbbf5c--