From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.ciao.gmane.io!not-for-mail From: Michael Albinus Newsgroups: gmane.emacs.devel Subject: Re: Remote asynchronous processes Date: Wed, 06 May 2020 13:59:34 +0200 Message-ID: <87a72ltgt5.fsf@gmx.de> References: <87tv1nwuv4.fsf@gmx.de> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Injection-Info: ciao.gmane.io; posting-host="ciao.gmane.io:159.69.161.202"; logging-data="44785"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.0.50 (gnu/linux) Cc: emacs-devel@gnu.org To: Philipp Stephani Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Wed May 06 14:03:45 2020 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1jWIm9-000BSf-5z for ged-emacs-devel@m.gmane-mx.org; Wed, 06 May 2020 14:03:45 +0200 Original-Received: from localhost ([::1]:50402 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jWIm7-0005LJ-TS for ged-emacs-devel@m.gmane-mx.org; Wed, 06 May 2020 08:03:44 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:54472) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jWIiC-0004N5-BQ for emacs-devel@gnu.org; Wed, 06 May 2020 07:59:40 -0400 Original-Received: from mout.gmx.net ([212.227.17.22]:60783) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jWIiA-0000WD-Ao for emacs-devel@gnu.org; Wed, 06 May 2020 07:59:39 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=badeba3b8450; t=1588766375; bh=AI0EHyecj4nZ2sLGDOH+sfndUuIQnLBfM+YTdUul0wQ=; h=X-UI-Sender-Class:From:To:Cc:Subject:References:Date:In-Reply-To; b=d31nP62XmGU50j9MiN4SAC17jq8wv3nQ5NppjEc0O67RvGegpgeQMdw6gFUVm+mFB K06F3EDklQze2P/6wnILX1LvNTvM3WuFfHuHRKAVJjk0i9kTDTkPrLzONOwEucggtT NMroqQ5wNgiT7E9zf4s9IzkzvFhPVLsMaYYNTLYQ= X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c Original-Received: from gandalf.gmx.de ([212.86.42.197]) by mail.gmx.com (mrgmx105 [212.227.17.168]) with ESMTPSA (Nemesis) id 1MtfJX-1jHf2W2Ehz-00v5jZ; Wed, 06 May 2020 13:59:35 +0200 In-Reply-To: <87tv1nwuv4.fsf@gmx.de> (Michael Albinus's message of "Mon, 13 Apr 2020 12:19:43 +0200") X-Provags-ID: V03:K1:jXN6/ffFXX+++YzZl0s17HJiARFm2B/onrwUngMeK3DK2O6/ne8 s4Cg9h3MIzBUdWS+xGZSDyu2YUz4sJWoXo09S8cLH0ja+CLWNSKfUICN2aEN4yHTXcK/qIh ZBfMYkHTkLm6crkT4aKFCTkC61zv21VRPmtxV0NeRl1HLCOMGvQIFDhHF0FW6Tj8fphyqIh Qvo69tZQNpSAg/4g1TFAw== X-UI-Out-Filterresults: notjunk:1;V03:K0:U+uGmXLKSHI=:j5XusAJfSct5SZI3HQ2NIu MiJzjXXXFuNKnD2fSlDZOFcPn+SHIcmojtpWFvBoF1XhTZ6yavPEituo4gOhpwdst7sEBuyCg Nkwx0Sg70rtkcJj77wehEd7/cbF5SnXbhUwz/Vl3LFMvg66lXX6SPoQ//YxNgL9eCqwwwwT0u WWGZKmoN8TUfh4cufVK4+7wOwBaW80S1PTFhrkDdJnZNkW44F+yL3AeWm9JAwyR+02cEr7aDq 2t282D4ydr4qFgAhkOYtYTy9+GsPqxllkwMVnZPtjguNXN9g4ET8IIpj/m51RVj3ErMViv1F/ Sq8mDU3yPFSV5wtzrAxoZ1QZyVjefe3MZz8d0LUXgOnZbNISJC+eyTCXbqq9eE0RzWXAb2Pis 4Auj2VfZf7y8FS4T+jCLe4zFAkAkLSlo1Ek4cQaeehBCxCCkYf6UxcanOQjZc64WmN+/FC6c0 9jIGixEL/ZRp4ymcb5vCJ/LgSbvmWmIr30/BIvNQ9b+mQpa3iWxKJL9gIddQYmGQbww2VP+RD g0FC+pYOC1TRnrWygBCBQ4X13bTYikfi0ANWp3eqf4F8diNv83A3ps03IfgU03Rv+2b+t82UP b/B1k+6ZTfBiZwmpkwp2wPu2xpshaavEOMOieDKzwNuSN+Roan4YnNXx/5Wv6wr9WyeyiiJbn Bm7iyRk2bV4f2zP1bni6RDeCNoAa9qpUo+2fFoXlx432fKlII0ojRO08CrmHuet5z8g0344aL A4eDd2hCrUosDo8r69FcvlbpYFOJ1AlqPWOpMdKP17WLg5zBiSILjOyxH/CMaeaC9pGIKvWH Received-SPF: pass client-ip=212.227.17.22; envelope-from=michael.albinus@gmx.de; helo=mout.gmx.net X-detected-operating-system: by eggs.gnu.org: First seen = 2020/05/06 07:59:36 X-ACL-Warn: Detected OS = Linux 2.2.x-3.x [generic] X-Spam_score_int: -25 X-Spam_score: -2.6 X-Spam_bar: -- X-Spam_report: (-2.6 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.io gmane.emacs.devel:249080 Archived-At: --=-=-= Content-Type: text/plain Michael Albinus writes: Hi, > On the Tramp ML, there is a discussion about performance of remote > asynchronous processes. start-file-process / make-process take too much > time to finish. > > One of the reasons is, that Tramp opens first a shell on the remote > host, performs sanity checks, and runs the command after that. Well, I > cannot change this in general; the sanity checks have been added due to > feedback from users. > > One idea to change the situation is, to remove all sanity checks from > make-process. That is, if a user has a default directory > "/ssh:user@host:/path/to/dir", and if he calls > > (make-process > :name "test" > :buffer (current-buffer) > :command '("cmd") > :file-handler t)) > > this is translated directly into > > ssh -l user -o ControlMaster=auto -o ControlPath='tramp.%C' \ > -o ControlPersist=no host "cd /path/to/dir; cmd" > > This would improve performance significantly. The drawback is, that > Tramp does not perform convenience checks, like password handling. I have played with this idea, and the output is the appended file tramp-make-process.el. It changes the make-process implementation of Tramp for all methods defined in tramp-sh.el (like "ssh") and tramp-adb.el. This is a proof-of-concept, and shouldn't be used for production. Read the commentary in the file for limitations. However, the speed optimization is remarkable. I've tested it with the following code snippet: --8<---------------cut here---------------start------------->8--- (let ((tramp-verbose 0) (default-directory "/ssh::/")) ;; Fill the caches. (start-file-process "" nil "true") ;; Run benchmark. (benchmark-run 10 (start-file-process "" nil "true"))) --8<---------------cut here---------------end--------------->8--- In the default case, the result is --8<---------------cut here---------------start------------->8--- (3.623666842 80 1.183512312) --8<---------------cut here---------------end--------------->8--- If tramp-make-process.el is loaded, the result is --8<---------------cut here---------------start------------->8--- (0.022762177 0 0.0) --8<---------------cut here---------------end--------------->8--- Similar results, if I use "/adb::/" as default directory: --8<---------------cut here---------------start------------->8--- (4.599374061 2 0.03429497299999973) --8<---------------cut here---------------end--------------->8--- vs --8<---------------cut here---------------start------------->8--- (0.013003183 0 0.0) --8<---------------cut here---------------end--------------->8--- Comments? Best regards, Michael. --=-=-= Content-Type: text/plain Content-Disposition: attachment; filename=tramp-make-process.el ;;; tramp-make-process.el --- Tramp alternative make-process -*- lexical-binding:t -*- ;; Copyright (C) 2020 Free Software Foundation, Inc. ;; Author: Michael Albinus ;; Keywords: comm, processes ;; Package: tramp ;; This file is part of GNU Emacs. ;; GNU Emacs is free software: you can redistribute it and/or modify ;; it under the terms of the GNU General Public License as published by ;; the Free Software Foundation, either version 3 of the License, or ;; (at your option) any later version. ;; GNU Emacs is distributed in the hope that it will be useful, ;; but WITHOUT ANY WARRANTY; without even the implied warranty of ;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the ;; GNU General Public License for more details. ;; You should have received a copy of the GNU General Public License ;; along with GNU Emacs. If not, see . ;;; Commentary: ;; An alternative implementation of `make-process' for methods in ;; tramp-sh.el and tramp-adb.el. It does not use shell commands for ;; execution of the asynchronous command. Instead, it calls the ;; command directly. This should result in a performance boost. ;; ;; Limitations of this approach: ;; ;; * It works only for connection methods defined in tramp-sh.el and ;; tramp-adb.el. ;; ;; * It does not support multi-hop methods. ;; ;; * It does not support user authentication, like password handling. ;; ;; * It does not support a separated error stream. ;; ;; * It cannot be killed via `interrupt-process'. ;; ;; * It does not report the remote terminal name via `process-tty-name'. ;; ;; * It does not set environment variable "INSIDE_EMACS". ;; ;; In order to gain even more performance, it is recommended to set or ;; bind `tramp-verbose' to 0 when running `make-process'. ;;; Code: ;; We use BUFFER also as connection buffer during setup. Because of ;; this, its original contents must be saved, and restored once ;; connection has been setup. (defun tramp-make-process (&rest args) "An alternative `make-process' implementation for Tramp files." (when args (with-parsed-tramp-file-name (expand-file-name default-directory) nil (let ((name (plist-get args :name)) (buffer (plist-get args :buffer)) (command (plist-get args :command)) (coding (plist-get args :coding)) (noquery (plist-get args :noquery)) (connection-type (plist-get args :connection-type)) (filter (plist-get args :filter)) (sentinel (plist-get args :sentinel)) (stderr (plist-get args :stderr))) (unless (stringp name) (signal 'wrong-type-argument (list #'stringp name))) (unless (or (null buffer) (bufferp buffer) (stringp buffer)) (signal 'wrong-type-argument (list #'stringp buffer))) (unless (consp command) (signal 'wrong-type-argument (list #'consp command))) (unless (or (null coding) (and (symbolp coding) (memq coding coding-system-list)) (and (consp coding) (memq (car coding) coding-system-list) (memq (cdr coding) coding-system-list))) (signal 'wrong-type-argument (list #'symbolp coding))) (unless (or (null connection-type) (memq connection-type '(pipe pty))) (signal 'wrong-type-argument (list #'symbolp connection-type))) (unless (or (null filter) (functionp filter)) (signal 'wrong-type-argument (list #'functionp filter))) (unless (or (null sentinel) (functionp sentinel)) (signal 'wrong-type-argument (list #'functionp sentinel))) (unless (or (null stderr) (bufferp stderr) (stringp stderr)) (signal 'wrong-type-argument (list #'stringp stderr))) (when (and (stringp stderr) (tramp-tramp-file-p stderr) (not (tramp-equal-remote default-directory stderr))) (signal 'file-error (list "Wrong stderr" stderr))) (let* ((buffer (if buffer (get-buffer-create buffer) ;; BUFFER can be nil. We use a temporary buffer. (generate-new-buffer tramp-temp-buffer-name))) (command (append `("cd" ,localname "&&") (mapcar #'tramp-shell-quote-argument command))) (bmp (and (buffer-live-p buffer) (buffer-modified-p buffer))) (name1 name) (i 0) ;; We do not want to raise an error when `make-process' ;; has been started several times in `eshell' and ;; friends. tramp-current-connection p) (while (get-process name1) ;; NAME must be unique as process name. (setq i (1+ i) name1 (format "%s<%d>" name i))) (setq name name1) ;; Set the new process properties. (tramp-set-connection-property v "process-name" name) (tramp-set-connection-property v "process-buffer" buffer) (with-current-buffer (tramp-get-connection-buffer v) (unwind-protect (let* ((login-program (or (tramp-get-method-parameter v 'tramp-login-program) "adb")) (login-args (or (tramp-get-method-parameter v 'tramp-login-args) '(("shell")))) (async-args (tramp-get-method-parameter v 'tramp-async-args)) ;; We don't create the temporary file. In ;; fact, it is just a prefix for the ;; ControlPath option of ssh; the real ;; temporary file has another name, and it is ;; created and protected by ssh. It is also ;; removed by ssh when the connection is ;; closed. The temporary file name is cached ;; in the main connection process, therefore ;; we cannot use `tramp-get-connection-process'. (tmpfile (with-tramp-connection-property (tramp-get-process v) "temp-file" (make-temp-name (expand-file-name tramp-temp-name-prefix (tramp-compat-temporary-file-directory))))) (options (tramp-ssh-controlmaster-options v)) spec) ;; Replace `login-args' place holders. (setq spec (format-spec-make ?t tmpfile) options (format-spec options spec) spec (format-spec-make ?h (or host "") ?u (or user "") ?p (or port "") ?c options ?l "") ;; Add arguments for asynchronous processes. login-args (append async-args login-args) ;; Expand format spec. login-args (tramp-compat-flatten-tree (mapcar (lambda (x) (setq x (mapcar (lambda (y) (format-spec y spec)) x)) (unless (member "" x) x)) login-args)) ;; Split ControlMaster options. login-args (tramp-compat-flatten-tree (mapcar (lambda (x) (split-string x " ")) login-args)) p (apply #'start-process name buffer login-program (append login-args command))) (tramp-message v 6 "%s" (string-join (process-command p) " ")) ;; Set sentinel and filter. (when sentinel (set-process-sentinel p sentinel)) (when filter (set-process-filter p filter)) ;; Set query flag and process marker for this ;; process. We ignore errors, because the ;; process could have finished already. (ignore-errors (set-process-query-on-exit-flag p (null noquery)) (set-marker (process-mark p) (point))) ;; We must flush them here already; otherwise ;; `rename-file', `delete-file' or ;; `insert-file-contents' will fail. (tramp-flush-connection-property v "process-name") (tramp-flush-connection-property v "process-buffer") ;; Return process. p) ;; Save exit. (if (string-match-p tramp-temp-buffer-name (buffer-name)) (ignore-errors (set-process-buffer p nil) (kill-buffer (current-buffer))) (set-buffer-modified-p bmp)) (tramp-flush-connection-property v "process-name") (tramp-flush-connection-property v "process-buffer")))))))) (with-eval-after-load 'tramp-adb (defalias 'tramp-adb-handle-make-process #'tramp-make-process)) (with-eval-after-load 'tramp-sh (defalias 'tramp-sh-handle-make-process #'tramp-make-process)) (add-hook 'tramp-unload-hook (lambda () (unload-feature 'tramp-make-process 'force))) (provide 'tramp-make-process) ;;; tramp-make-process.el ends here --=-=-=--