From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!not-for-mail From: Michael Albinus Newsgroups: gmane.emacs.devel,gmane.emacs.pretest.bugs Subject: Re: 22.1.50; insert-file-contents is slow under tramp Date: Sun, 26 Aug 2007 12:36:32 +0200 Message-ID: <871wdqlgzz.fsf@gmx.de> References: <87veb6l9b0.fsf@escher.local.home> <46CD98DC.1050701@gmx.at> <87r6lul2fc.fsf@escher.local.home> <46CDC8B3.7000301@gmx.at> <46CDED2B.3080904@gmx.at> <87k5rmkmu0.fsf@escher.local.home> <46CDF5B7.2030201@gmx.at> <87fy2akjzt.fsf@escher.local.home> <87bqcxzeat.fsf@gmx.de> NNTP-Posting-Host: lo.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Trace: sea.gmane.org 1188124621 27055 80.91.229.12 (26 Aug 2007 10:37:01 GMT) X-Complaints-To: usenet@sea.gmane.org NNTP-Posting-Date: Sun, 26 Aug 2007 10:37:01 +0000 (UTC) Cc: emacs-pretest-bug@gnu.org, Stephen Berman , emacs-devel@gnu.org To: Stefan Monnier Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Sun Aug 26 12:37:00 2007 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([199.232.76.165]) by lo.gmane.org with esmtp (Exim 4.50) id 1IPFUI-0003AI-BT for ged-emacs-devel@m.gmane.org; Sun, 26 Aug 2007 12:36:58 +0200 Original-Received: from localhost ([127.0.0.1] helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1IPFUH-00089O-PC for ged-emacs-devel@m.gmane.org; Sun, 26 Aug 2007 06:36:57 -0400 Original-Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1IPFTy-00082d-S8 for emacs-devel@gnu.org; Sun, 26 Aug 2007 06:36:38 -0400 Original-Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1IPFTx-00082M-Na for emacs-devel@gnu.org; Sun, 26 Aug 2007 06:36:38 -0400 Original-Received: from [199.232.76.173] (helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1IPFTx-00082E-Cm for emacs-devel@gnu.org; Sun, 26 Aug 2007 06:36:37 -0400 Original-Received: from mail.gmx.net ([213.165.64.20]) by monty-python.gnu.org with smtp (Exim 4.60) (envelope-from ) id 1IPFTw-0003Sy-Pt for emacs-devel@gnu.org; Sun, 26 Aug 2007 06:36:37 -0400 Original-Received: (qmail invoked by alias); 26 Aug 2007 10:36:35 -0000 Original-Received: from p57A238F8.dip0.t-ipconnect.de (EHLO localhost.local) [87.162.56.248] by mail.gmx.net (mp018) with SMTP; 26 Aug 2007 12:36:35 +0200 X-Authenticated: #3708877 X-Provags-ID: V01U2FsdGVkX1/VA990bp3Vx0RmNGmxacOFVu+P5fqd6KwvoYoBjh gpjG/oVbU5bK4m User-Agent: Gnus/5.110007 (No Gnus v0.7) Emacs/22.1.50 (gnu/linux) X-Y-GMX-Trusted: 0 X-Detected-Kernel: Linux 2.6, seldom 2.4 (older, 4) X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Xref: news.gmane.org gmane.emacs.devel:77204 gmane.emacs.pretest.bugs:19631 Archived-At: Stefan Monnier writes: >> Yes. I've installed a patch in the trunk, which makes a fast track for >> "remote" files being on localhost. > > How does that work? I mean when accessing files on the localhost with > Tramp, the most likely situation I can think of is the `su' method so how > what does your "fast track" do to transfer the file faster? Does it copy it > to /tmp at the "remote" end, then make it world-readable, then copy it on > the "local" end, then remove it at the remote end? Similar approach. It calls `tramp-do-copy-or-rename-file-directly', which copies the "remote" file to /tmp, and assigns proper permissions. Then `insert-file-contents' for the _local_ temporary file is applied, which shows the file in the buffer, and the temporary file is deleted. >> For real remote hosts, I recommend to use scp when accessing huge files. > > Reminds me: where is the bottleneck in the inline method? I mean > fundamentally doing a base64 encoding/decoding shouldn't cost that much > (negligible CPU-wise and a mere 33% in network transfer). Maybe we could > attack this part to improve performance? I haven't investigated too much. mimencode and base64-decode are pretty fast, even on my stone-age machine, with an 8MB sized file: | (with-current-buffer (get-buffer-create "foo") | (elp-instrument-function 'call-process) | (elp-instrument-function 'base64-decode-region) | (erase-buffer) | | (call-process "mimencode" nil t nil "/var/cache/apt/pkgcache.bin") | (base64-decode-region (point-min) (point-max)) | (elp-results)) | | Function Name Call Count Elapsed Time Average Time | ==================== ========== ============ ============ | base64-decode-region 1 7.732203 7.732203 | call-process 1 1.917887 1.917887 But with the installed "fast track", it is remarkable faster than even these two basic functions: | (with-current-buffer (get-buffer-create "foo") | (elp-instrument-function 'insert-file-contents) | (erase-buffer) | (setq tramp-verbose 0) | | ;; This opens the connection, including password handover etc | (file-attributes "/sudo::") | | (insert-file-contents "/sudo::/var/cache/apt/pkgcache.bin") | (elp-results)) | | Function Name Call Count Elapsed Time Average Time | ==================== ========== ============ ============ | insert-file-contents 2 1.64688 0.82344 Without the fast track, the access time is inacceptable large. I suspect that `tramp-wait-for-output' is the bottleneck, because it must check for the resulting shell prompt, handle echoing shells, etc. I will investigate next days whether there can be an improvement. > Stefan Best regards, Michael.