From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED!not-for-mail From: Gemini Lasswell Newsgroups: gmane.emacs.bugs Subject: bug#28227: 26.0.50; Tramp tests are slow Date: Mon, 04 Sep 2017 15:43:17 -0700 Message-ID: <87a82ankze.fsf@runbox.com> References: <878ti81xwb.fsf@chinook> <87lgm8dqev.fsf@detlef> <874lsok15a.fsf@runbox.com> <87lgm079ra.fsf@users.sourceforge.net> <87inh3iym8.fsf@runbox.com> <877exili1a.fsf@detlef> NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Trace: blaine.gmane.org 1504565136 3179 195.159.176.226 (4 Sep 2017 22:45:36 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Mon, 4 Sep 2017 22:45:36 +0000 (UTC) User-Agent: mu4e 0.9.18; emacs 25.1.1 Cc: 28227@debbugs.gnu.org, npostavs@users.sourceforge.net To: Michael Albinus Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Tue Sep 05 00:45:21 2017 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dp07G-0008H9-0A for geb-bug-gnu-emacs@m.gmane.org; Tue, 05 Sep 2017 00:45:14 +0200 Original-Received: from localhost ([::1]:55931 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dp07N-0008UV-1w for geb-bug-gnu-emacs@m.gmane.org; Mon, 04 Sep 2017 18:45:21 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:59272) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dp079-0008MJ-Tt for bug-gnu-emacs@gnu.org; Mon, 04 Sep 2017 18:45:13 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dp074-0002Gk-EE for bug-gnu-emacs@gnu.org; Mon, 04 Sep 2017 18:45:07 -0400 Original-Received: from debbugs.gnu.org ([208.118.235.43]:40896) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1dp074-0002Ge-32 for bug-gnu-emacs@gnu.org; Mon, 04 Sep 2017 18:45:02 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1dp073-0007Sh-LX for bug-gnu-emacs@gnu.org; Mon, 04 Sep 2017 18:45:01 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Gemini Lasswell Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Mon, 04 Sep 2017 22:45:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 28227 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: Original-Received: via spool by 28227-submit@debbugs.gnu.org id=B28227.150456506128617 (code B ref 28227); Mon, 04 Sep 2017 22:45:01 +0000 Original-Received: (at 28227) by debbugs.gnu.org; 4 Sep 2017 22:44:21 +0000 Original-Received: from localhost ([127.0.0.1]:49575 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dp06O-0007RV-Mh for submit@debbugs.gnu.org; Mon, 04 Sep 2017 18:44:21 -0400 Original-Received: from aibo.runbox.com ([91.220.196.211]:40756) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dp06M-0007RH-R7 for 28227@debbugs.gnu.org; Mon, 04 Sep 2017 18:44:20 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=runbox.com; s=rbselector1; h=Content-Type:MIME-Version:Message-ID:Date:In-reply-to: Subject:Cc:To:From:References; bh=R8UkBZvJUB687mFOnxgOVRXDgLCPFudUmFkFP3gusbU=; b=EM4kPbt+RDU0hr3bnUOZnxME9+ 2WvJz0Ou1+tbFHEKDh4u+vlJXgV8uqqGyo0gjUOj/vDyQYbBHP8cFDIzC399AGK6aoTaaW6X1bljQ yykuLrBM/HC/2DXWqJlEXaPHn04Ef6eF6WmbYyg/fpQ9QKWsxR+PtGGLjxqtfNhzYJwx8wrOyOpRz tLobWdUdrvlgiotXbenZQVndmQofDTFYvtAf2sA8Z6KM8lW25kpFbhI52oP8tv49VAMHNmddWR8pA EtqfW/IkcyNq3VTp3mS+pXO0Amqy6Cwmz/HMuRI9Micra63J/xIxCzbdyNThBWHPVpoXQe8f7W04M lYhPe8gQ==; Original-Received: from [10.9.9.211] (helo=mailfront11.runbox.com) by mailtransmit03.runbox with esmtp (Exim 4.86_2) (envelope-from ) id 1dp06G-0006vu-FC; Tue, 05 Sep 2017 00:44:12 +0200 Original-Received: from c-24-22-244-161.hsd1.wa.comcast.net ([24.22.244.161] helo=chinook) by mailfront11.runbox.com with esmtpsa (uid:179284 ) (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) id 1dp05Y-0000Kn-6M; Tue, 05 Sep 2017 00:43:28 +0200 In-reply-to: <877exili1a.fsf@detlef> X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 208.118.235.43 X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.org gmane.emacs.bugs:136587 Archived-At: --=-=-= Content-Type: text/plain Michael Albinus writes: > Looks good to me, thanks a lot! You could apply this to master. Hi Michael, Here's a new version of the patch which incorporates your suggestions: --=-=-= Content-Type: text/plain Content-Disposition: attachment; filename=0001-Reduce-Tramp-s-memory-usage.patch >From 41f70cc2b89ac78f6b314cf140a4e7a9250c8eb2 Mon Sep 17 00:00:00 2001 From: Gemini Lasswell Date: Wed, 30 Aug 2017 07:11:41 -0700 Subject: [PATCH] Reduce Tramp's memory usage Construct Tramp syntax strings and regular expressions once instead of every time they are used, and store them in alists keyed by Tramp syntax. * tramp.el (tramp-build-remote-file-name-spec-regexp) (tramp-build-file-name-structure): New functions. (tramp-prefix-format-alist, tramp-prefix-regexp-alist) (tramp-method-regexp-alist) (tramp-postfix-method-format-alist) (tramp-postfix-method-regexp-alist) (tramp-prefix-ipv6-format-alist, tramp-prefix-ipv6-regexp-alist) (tramp-postfix-ipv6-format-alist) (tramp-postfix-ipv6-regexp-alist) (tramp-postfix-host-format-alist) (tramp-postfix-host-regexp-alist) (tramp-remote-file-name-spec-regexp-alist) (tramp-file-name-structure-alist): New constants. (tramp-lookup-syntax): New function. (tramp-prefix-format, tramp-prefix-regexp, tramp-method-regexp) (tramp-postfix-method-format, tramp-postfix-method-regexp) (tramp-prefix-ipv6-format, tramp-prefix-ipv6-regexp) (tramp-postfix-ipv6-format, tramp-postfix-ipv6-regexp) (tramp-postfix-host-format, tramp-postfix-host-regexp) (tramp-remote-file-name-spec-regexp, tramp-file-name-structure): Use it. --- lisp/net/tramp.el | 172 ++++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 127 insertions(+), 45 deletions(-) diff --git a/lisp/net/tramp.el b/lisp/net/tramp.el index 1a5cda7e20..adc59cb542 100644 --- a/lisp/net/tramp.el +++ b/lisp/net/tramp.el @@ -700,40 +700,69 @@ tramp-syntax-values (setq values (mapcar 'last values) values (mapcar 'car values)))) +(defun tramp-lookup-syntax (alist) + "Look up a syntax string in ALIST according to `tramp-compat-tramp-syntax.' +Raise an error if `tramp-syntax' is invalid." + (or (cdr (assq (tramp-compat-tramp-syntax) alist)) + (error "Wrong `tramp-syntax' %s" tramp-syntax))) + +(defconst tramp-prefix-format-alist + '((default . "/") + (simplified . "/") + (separate . "/[")) + "Alist mapping Tramp syntax to strings beginning Tramp file names.") + (defun tramp-prefix-format () "String matching the very beginning of Tramp file names. Used in `tramp-make-tramp-file-name'." - (cond ((eq (tramp-compat-tramp-syntax) 'default) "/") - ((eq (tramp-compat-tramp-syntax) 'simplified) "/") - ((eq (tramp-compat-tramp-syntax) 'separate) "/[") - (t (error "Wrong `tramp-syntax' %s" tramp-syntax)))) + (tramp-lookup-syntax tramp-prefix-format-alist)) + +(defconst tramp-prefix-regexp-alist + (mapcar (lambda (x) + (cons (car x) (concat "^" (regexp-quote (cdr x))))) + tramp-prefix-format-alist) + "Alist of regexps matching the beginnings of Tramp file names. +Keyed by Tramp syntax. Derived from `tramp-prefix-format-alist'.") (defun tramp-prefix-regexp () "Regexp matching the very beginning of Tramp file names. Should always start with \"^\". Derived from `tramp-prefix-format'." - (concat "^" (regexp-quote (tramp-prefix-format)))) + (tramp-lookup-syntax tramp-prefix-regexp-alist)) + +(defconst tramp-method-regexp-alist + '((default . "[a-zA-Z0-9-]+") + (simplified . "") + (separate . "[a-zA-Z0-9-]*")) + "Alist mapping Tramp syntax to regexps matching methods identifiers.") (defun tramp-method-regexp () "Regexp matching methods identifiers. The `ftp' syntax does not support methods." - (cond ((eq (tramp-compat-tramp-syntax) 'default) "[a-zA-Z0-9-]+") - ((eq (tramp-compat-tramp-syntax) 'simplified) "") - ((eq (tramp-compat-tramp-syntax) 'separate) "[a-zA-Z0-9-]*") - (t (error "Wrong `tramp-syntax' %s" tramp-syntax)))) + (tramp-lookup-syntax tramp-method-regexp-alist)) + +(defconst tramp-postfix-method-format-alist + '((default . ":") + (simplified . "") + (separate . "/")) + "Alist mapping Tramp syntax to the delimiter after the method.") (defun tramp-postfix-method-format () "String matching delimiter between method and user or host names. The `ftp' syntax does not support methods. Used in `tramp-make-tramp-file-name'." - (cond ((eq (tramp-compat-tramp-syntax) 'default) ":") - ((eq (tramp-compat-tramp-syntax) 'simplified) "") - ((eq (tramp-compat-tramp-syntax) 'separate) "/") - (t (error "Wrong `tramp-syntax' %s" tramp-syntax)))) + (tramp-lookup-syntax tramp-postfix-method-format-alist)) + +(defconst tramp-postfix-method-regexp-alist + (mapcar (lambda (x) + (cons (car x) (regexp-quote (cdr x)))) + tramp-postfix-method-format-alist) + "Alist mapping Tramp syntax to regexp matching delimiter after method. +Derived from `tramp-postfix-method-format-alist'.") (defun tramp-postfix-method-regexp () "Regexp matching delimiter between method and user or host names. Derived from `tramp-postfix-method-format'." - (regexp-quote (tramp-postfix-method-format))) + (tramp-lookup-syntax tramp-postfix-method-regexp-alist)) (defconst tramp-user-regexp "[^/|: \t]+" "Regexp matching user names.") @@ -769,18 +798,28 @@ tramp-postfix-user-regexp (defconst tramp-host-regexp "[a-zA-Z0-9_.-]+" "Regexp matching host names.") +(defconst tramp-prefix-ipv6-format-alist + '((default . "[") + (simplified . "[") + (separate . "")) + "Alist mapping Tramp syntax to strings prefixing IPv6 addresses.") + (defun tramp-prefix-ipv6-format () "String matching left hand side of IPv6 addresses. Used in `tramp-make-tramp-file-name'." - (cond ((eq (tramp-compat-tramp-syntax) 'default) "[") - ((eq (tramp-compat-tramp-syntax) 'simplified) "[") - ((eq (tramp-compat-tramp-syntax) 'separate) "") - (t (error "Wrong `tramp-syntax' %s" tramp-syntax)))) + (tramp-lookup-syntax tramp-prefix-ipv6-format-alist)) + +(defconst tramp-prefix-ipv6-regexp-alist + (mapcar (lambda (x) + (cons (car x) (regexp-quote (cdr x)))) + tramp-prefix-ipv6-format-alist) + "Alist mapping Tramp syntax to regexp matching prefix of IPv6 addresses. +Derived from `tramp-prefix-ipv6-format-alist'") (defun tramp-prefix-ipv6-regexp () "Regexp matching left hand side of IPv6 addresses. Derived from `tramp-prefix-ipv6-format'." - (regexp-quote (tramp-prefix-ipv6-format))) + (tramp-lookup-syntax tramp-prefix-ipv6-regexp-alist)) ;; The following regexp is a bit sloppy. But it shall serve our ;; purposes. It covers also IPv4 mapped IPv6 addresses, like in @@ -789,18 +828,28 @@ tramp-ipv6-regexp "\\(?:\\(?:[a-zA-Z0-9]+\\)?:\\)+[a-zA-Z0-9.]+" "Regexp matching IPv6 addresses.") +(defconst tramp-postfix-ipv6-format-alist + '((default . "]") + (simplified . "]") + (separate . "")) + "Alist mapping Tramp syntax to suffix for IPv6 addresses.") + (defun tramp-postfix-ipv6-format () "String matching right hand side of IPv6 addresses. Used in `tramp-make-tramp-file-name'." - (cond ((eq (tramp-compat-tramp-syntax) 'default) "]") - ((eq (tramp-compat-tramp-syntax) 'simplified) "]") - ((eq (tramp-compat-tramp-syntax) 'separate) "") - (t (error "Wrong `tramp-syntax' %s" tramp-syntax)))) + (tramp-lookup-syntax tramp-postfix-ipv6-format-alist)) + +(defconst tramp-postfix-ipv6-regexp-alist + (mapcar (lambda (x) + (cons (car x) (regexp-quote (cdr x)))) + tramp-postfix-ipv6-format-alist) + "Alist mapping Tramp syntax to regexps matching IPv6 suffixes. +Derived from `tramp-postfix-ipv6-format-alist'.") (defun tramp-postfix-ipv6-regexp () "Regexp matching right hand side of IPv6 addresses. Derived from `tramp-postfix-ipv6-format'." - (regexp-quote (tramp-postfix-ipv6-format))) + (tramp-lookup-syntax tramp-postfix-ipv6-format-alist)) (defconst tramp-prefix-port-format "#" "String matching delimiter between host names and port numbers.") @@ -827,18 +876,28 @@ tramp-postfix-hop-regexp "Regexp matching delimiter after ad-hoc hop definitions. Derived from `tramp-postfix-hop-format'.") +(defconst tramp-postfix-host-format-alist + '((default . ":") + (simplified . ":") + (separate . "]")) + "Alist mapping Tramp syntax to strings between host and local names.") + (defun tramp-postfix-host-format () "String matching delimiter between host names and localnames. Used in `tramp-make-tramp-file-name'." - (cond ((eq (tramp-compat-tramp-syntax) 'default) ":") - ((eq (tramp-compat-tramp-syntax) 'simplified) ":") - ((eq (tramp-compat-tramp-syntax) 'separate) "]") - (t (error "Wrong `tramp-syntax' %s" tramp-syntax)))) + (tramp-lookup-syntax tramp-postfix-host-format-alist)) + +(defconst tramp-postfix-host-regexp-alist + (mapcar (lambda (x) + (cons (car x) (regexp-quote (cdr x)))) + tramp-postfix-host-format-alist) + "Alist mapping Tramp syntax to regexp matching name delimiters. +Derived from `tramp-postfix-host-format-alist'.") (defun tramp-postfix-host-regexp () "Regexp matching delimiter between host names and localnames. Derived from `tramp-postfix-host-format'." - (regexp-quote (tramp-postfix-host-format))) + (tramp-lookup-syntax tramp-postfix-host-regexp-alist)) (defconst tramp-localname-regexp ".*$" "Regexp matching localnames.") @@ -851,16 +910,46 @@ tramp-unknown-id-integer ;;; File name format: +(defun tramp-build-remote-file-name-spec-regexp (syntax) + "Construct a regexp matching a Tramp file name for a Tramp SYNTAX." + (let ((tramp-syntax syntax)) + (concat + "\\(" (tramp-method-regexp) "\\)" (tramp-postfix-method-regexp) + "\\(?:" "\\(" tramp-user-regexp "\\)" tramp-postfix-user-regexp "\\)?" + "\\(" "\\(?:" tramp-host-regexp "\\|" + (tramp-prefix-ipv6-regexp) + "\\(?:" tramp-ipv6-regexp "\\)?" + (tramp-postfix-ipv6-regexp) "\\)?" + "\\(?:" tramp-prefix-port-regexp tramp-port-regexp "\\)?" "\\)?"))) + +(defconst tramp-remote-file-name-spec-regexp-alist + `((default . ,(tramp-build-remote-file-name-spec-regexp 'default)) + (simplified . ,(tramp-build-remote-file-name-spec-regexp 'simplified)) + (separate . ,(tramp-build-remote-file-name-spec-regexp 'separate))) + "Alist mapping Tramp syntax to regexps matching Tramp file names.") + (defun tramp-remote-file-name-spec-regexp () "Regular expression matching a Tramp file name between prefix and postfix." - (concat - "\\(" (tramp-method-regexp) "\\)" (tramp-postfix-method-regexp) - "\\(?:" "\\(" tramp-user-regexp "\\)" tramp-postfix-user-regexp "\\)?" - "\\(" "\\(?:" tramp-host-regexp "\\|" - (tramp-prefix-ipv6-regexp) - "\\(?:" tramp-ipv6-regexp "\\)?" - (tramp-postfix-ipv6-regexp) "\\)?" - "\\(?:" tramp-prefix-port-regexp tramp-port-regexp "\\)?" "\\)?")) + (tramp-lookup-syntax tramp-remote-file-name-spec-regexp-alist)) + +(defun tramp-build-file-name-structure (syntax) + "Construct the Tramp file name structure for SYNTAX. +See `tramp-file-name-structure'." + (let ((tramp-syntax syntax)) + (list + (concat + (tramp-prefix-regexp) + "\\(" "\\(?:" (tramp-remote-file-name-spec-regexp) + tramp-postfix-hop-regexp "\\)+" "\\)?" + (tramp-remote-file-name-spec-regexp) (tramp-postfix-host-regexp) + "\\(" tramp-localname-regexp "\\)") + 5 6 7 8 1))) + +(defconst tramp-file-name-structure-alist + `((default . ,(tramp-build-file-name-structure 'default)) + (simplified . ,(tramp-build-file-name-structure 'simplified)) + (separate . ,(tramp-build-file-name-structure 'separate))) + "Alist mapping Tramp syntax to the file name structure for that syntax.") (defun tramp-file-name-structure () "List of six elements (REGEXP METHOD USER HOST FILE HOP), detailing \ @@ -881,14 +970,7 @@ tramp-file-name-structure means the opening parentheses are counted to identify the pair. See also `tramp-file-name-regexp'." - (list - (concat - (tramp-prefix-regexp) - "\\(" "\\(?:" (tramp-remote-file-name-spec-regexp) - tramp-postfix-hop-regexp "\\)+" "\\)?" - (tramp-remote-file-name-spec-regexp) (tramp-postfix-host-regexp) - "\\(" tramp-localname-regexp "\\)") - 5 6 7 8 1)) + (tramp-lookup-syntax tramp-file-name-structure-alist)) (defun tramp-file-name-regexp () "Regular expression matching file names handled by Tramp. -- 2.14.1 --=-=-= Content-Type: text/plain I submitted a request for write permissions this weekend, so I'll push it after I hear back about that, if there's no other feedback. Best, Gemini --=-=-=--