From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED!not-for-mail From: Gemini Lasswell Newsgroups: gmane.emacs.bugs Subject: bug#28227: 26.0.50; Tramp tests are slow Date: Thu, 31 Aug 2017 07:46:33 -0700 Message-ID: <87inh3iym8.fsf@runbox.com> References: <878ti81xwb.fsf@chinook> <87lgm8dqev.fsf@detlef> <874lsok15a.fsf@runbox.com> <87lgm079ra.fsf@users.sourceforge.net> NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Trace: blaine.gmane.org 1504190904 23820 195.159.176.226 (31 Aug 2017 14:48:24 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Thu, 31 Aug 2017 14:48:24 +0000 (UTC) User-Agent: mu4e 0.9.18; emacs 25.1.1 Cc: Michael Albinus , 28227@debbugs.gnu.org To: npostavs@users.sourceforge.net Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Thu Aug 31 16:48:13 2017 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dnQl1-0004dh-S6 for geb-bug-gnu-emacs@m.gmane.org; Thu, 31 Aug 2017 16:47:48 +0200 Original-Received: from localhost ([::1]:56170 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dnQl9-0000VA-1C for geb-bug-gnu-emacs@m.gmane.org; Thu, 31 Aug 2017 10:47:55 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:50520) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dnQkN-0008Qc-Ox for bug-gnu-emacs@gnu.org; Thu, 31 Aug 2017 10:47:13 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dnQkI-00084P-6D for bug-gnu-emacs@gnu.org; Thu, 31 Aug 2017 10:47:07 -0400 Original-Received: from debbugs.gnu.org ([208.118.235.43]:57241) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1dnQkH-00084A-TS for bug-gnu-emacs@gnu.org; Thu, 31 Aug 2017 10:47:02 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1dnQkH-0004sq-MS for bug-gnu-emacs@gnu.org; Thu, 31 Aug 2017 10:47:01 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Gemini Lasswell Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Thu, 31 Aug 2017 14:47:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 28227 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: Original-Received: via spool by 28227-submit@debbugs.gnu.org id=B28227.150419082018765 (code B ref 28227); Thu, 31 Aug 2017 14:47:01 +0000 Original-Received: (at 28227) by debbugs.gnu.org; 31 Aug 2017 14:47:00 +0000 Original-Received: from localhost ([127.0.0.1]:37689 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dnQkG-0004sb-2U for submit@debbugs.gnu.org; Thu, 31 Aug 2017 10:47:00 -0400 Original-Received: from aibo.runbox.com ([91.220.196.211]:58084) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dnQkD-0004sQ-NJ for 28227@debbugs.gnu.org; Thu, 31 Aug 2017 10:46:58 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=runbox.com; s=rbselector1; h=Content-Type:MIME-Version:Date:In-reply-to:Message-ID: Subject:Cc:To:From:References; bh=drgOyQ3gle/vhjDJrYak8BkAghxStyUZuDUKcOSeiA4=; b=bBI+m/vJmJ/rBHzKP9Qq8fVGS6 sLISGbUwIJ3uFNOfLEXpm2Iklpb1jgCZiBkC5lJlM0WJWzIL7jY4Skf++Rqr6qqrFkBBTleq5vxtV ihO+TiTVLCNECBLbXqc8JN+NF6l42V0yQ9BFQpz1FXTgImddsIil01HMCYkfbvjBIICwD/S6tioyJ K1y1eB9p56w3YS9J0+vKuHSP6jspRILquM5EEgP2yIPsa2ydTIi/EY4ZJR/LVBvFyev/ute++Rf9n 9yWQIxtP9Zuq/gFXDwce3Qb1buq2B7l6zPtj8fwEacd3RBHiyXwx6fes9zTw4l0ACkv3F7ggANjAi qg+ruc5A==; Original-Received: from [10.9.9.210] (helo=mailfront10.runbox.com) by mailtransmit02.runbox with esmtp (Exim 4.86_2) (envelope-from ) id 1dnQk9-0000T1-2d; Thu, 31 Aug 2017 16:46:53 +0200 Original-Received: from h193.31.25.72.ip.windstream.net ([72.25.31.193] helo=sockeye) by mailfront10.runbox.com with esmtpsa (uid:179284 ) (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) id 1dnQjx-00026h-Jm; Thu, 31 Aug 2017 16:46:42 +0200 In-reply-to: <87lgm079ra.fsf@users.sourceforge.net> X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 208.118.235.43 X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.org gmane.emacs.bugs:136406 Archived-At: --=-=-= Content-Type: text/plain Here's a new patch which uses defun and doesn't have the let binding. Performance is not noticably different. --=-=-= Content-Type: text/plain Content-Disposition: attachment; filename=0001-Reduce-Tramp-s-memory-usage.patch >From ac19e9b86c812e0e6cc97a950c0ee87f6739ab47 Mon Sep 17 00:00:00 2001 From: Gemini Lasswell Date: Wed, 30 Aug 2017 07:11:41 -0700 Subject: [PATCH] Reduce Tramp's memory usage Construct Tramp syntax strings and regular expressions once instead of every time they are used, and store them in alists keyed by Tramp syntax. * tramp.el (tramp-build-remote-file-name-spec-regexp) (tramp-build-file-name-structure): New functions. (tramp-prefix-format-alist, tramp-prefix-regexp-alist) (tramp-prefix-method-regexp-alist) (tramp-postfix-method-format-alist) (tramp-postfix-method-regexp-alist) (tramp-prefix-ipv6-format-alist, tramp-prefix-ipv6-regexp-alist) (tramp-postfix-ipv6-format-alist) (tramp-postfix-ipv6-regexp-alist) (tramp-postfix-host-format-alist) (tramp-postfix-host-regexp-alist) (tramp-remote-file-name-spec-regexp-alist) (tramp-file-name-structure-alist): New constants. (tramp-lookup-syntax): New function. (tramp-prefix-format, tramp-prefix-regexp, tramp-method-regexp) (tramp-postfix-method-format, tramp-postfix-method-regexp) (tramp-prefix-ipv6-format, tramp-prefix-ipv6-regexp) (tramp-postfix-ipv6-format, tramp-postfix-ipv6-regexp) (tramp-postfix-host-format, tramp-postfix-host-regexp) (tramp-remote-file-name-spec-regexp, tramp-file-name-structure): Use it. --- lisp/net/tramp.el | 172 ++++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 127 insertions(+), 45 deletions(-) diff --git a/lisp/net/tramp.el b/lisp/net/tramp.el index 1a5cda7e20..06a69044e0 100644 --- a/lisp/net/tramp.el +++ b/lisp/net/tramp.el @@ -700,40 +700,69 @@ tramp-syntax-values (setq values (mapcar 'last values) values (mapcar 'car values)))) +(defun tramp-lookup-syntax (alist) + "Look up a syntax string in ALIST according to `tramp-compat-tramp-syntax.' +Raise an error if `tramp-syntax' is invalid." + (or (cdr (assq (tramp-compat-tramp-syntax) alist)) + (error "Wrong `tramp-syntax' %s" tramp-syntax))) + +(defconst tramp-prefix-format-alist + '((default . "/") + (simplified . "/") + (separate . "/[")) + "Alist mapping Tramp syntax to strings beginning Tramp file names.") + (defun tramp-prefix-format () "String matching the very beginning of Tramp file names. Used in `tramp-make-tramp-file-name'." - (cond ((eq (tramp-compat-tramp-syntax) 'default) "/") - ((eq (tramp-compat-tramp-syntax) 'simplified) "/") - ((eq (tramp-compat-tramp-syntax) 'separate) "/[") - (t (error "Wrong `tramp-syntax' %s" tramp-syntax)))) + (tramp-lookup-syntax tramp-prefix-format-alist)) + +(defconst tramp-prefix-regexp-alist + `((default . ,(concat "^" (regexp-quote "/"))) + (simplified . ,(concat "^" (regexp-quote "/"))) + (separate . ,(concat "^" (regexp-quote "/[")))) + "Alist of regexps matching the beginnings of Tramp file names. +Keyed by Tramp syntax. Derived from `tramp-prefix-format-alist'.") (defun tramp-prefix-regexp () "Regexp matching the very beginning of Tramp file names. Should always start with \"^\". Derived from `tramp-prefix-format'." - (concat "^" (regexp-quote (tramp-prefix-format)))) + (tramp-lookup-syntax tramp-prefix-regexp-alist)) + +(defconst tramp-prefix-method-regexp-alist + '((default . "[a-zA-Z0-9-]+") + (simplified . "") + (separate . "[a-zA-Z0-9-]*")) + "Alist mapping Tramp syntax to regexps matching methods identifiers.") (defun tramp-method-regexp () "Regexp matching methods identifiers. The `ftp' syntax does not support methods." - (cond ((eq (tramp-compat-tramp-syntax) 'default) "[a-zA-Z0-9-]+") - ((eq (tramp-compat-tramp-syntax) 'simplified) "") - ((eq (tramp-compat-tramp-syntax) 'separate) "[a-zA-Z0-9-]*") - (t (error "Wrong `tramp-syntax' %s" tramp-syntax)))) + (tramp-lookup-syntax tramp-prefix-method-regexp-alist)) + +(defconst tramp-postfix-method-format-alist + '((default . ":") + (simplified . "") + (separate . "/")) + "Alist mapping Tramp syntax to the delimiter after the method.") (defun tramp-postfix-method-format () "String matching delimiter between method and user or host names. The `ftp' syntax does not support methods. Used in `tramp-make-tramp-file-name'." - (cond ((eq (tramp-compat-tramp-syntax) 'default) ":") - ((eq (tramp-compat-tramp-syntax) 'simplified) "") - ((eq (tramp-compat-tramp-syntax) 'separate) "/") - (t (error "Wrong `tramp-syntax' %s" tramp-syntax)))) + (tramp-lookup-syntax tramp-postfix-method-format-alist)) + +(defconst tramp-postfix-method-regexp-alist + `((default . ,(regexp-quote ":")) + (simplified . ,(regexp-quote "")) + (separate . ,(regexp-quote "/"))) + "Alist mapping Tramp syntax to regexp matching delimiter after method. +Derived from `tramp-postfix-method-format-alist'.") (defun tramp-postfix-method-regexp () "Regexp matching delimiter between method and user or host names. Derived from `tramp-postfix-method-format'." - (regexp-quote (tramp-postfix-method-format))) + (tramp-lookup-syntax tramp-postfix-method-regexp-alist)) (defconst tramp-user-regexp "[^/|: \t]+" "Regexp matching user names.") @@ -769,18 +798,28 @@ tramp-postfix-user-regexp (defconst tramp-host-regexp "[a-zA-Z0-9_.-]+" "Regexp matching host names.") +(defconst tramp-prefix-ipv6-format-alist + '((default . "[") + (simplified . "[") + (separate . "")) + "Alist mapping Tramp syntax to strings prefixing IPv6 addresses.") + (defun tramp-prefix-ipv6-format () "String matching left hand side of IPv6 addresses. Used in `tramp-make-tramp-file-name'." - (cond ((eq (tramp-compat-tramp-syntax) 'default) "[") - ((eq (tramp-compat-tramp-syntax) 'simplified) "[") - ((eq (tramp-compat-tramp-syntax) 'separate) "") - (t (error "Wrong `tramp-syntax' %s" tramp-syntax)))) + (tramp-lookup-syntax tramp-prefix-ipv6-format-alist)) + +(defconst tramp-prefix-ipv6-regexp-alist + `((default . ,(regexp-quote "[")) + (simplified . ,(regexp-quote "[")) + (separate . ,(regexp-quote ""))) + "Alist mapping Tramp syntax to regexp matching prefix of IPv6 addresses. +Derived from `tramp-prefix-ipv6-format-alist'") (defun tramp-prefix-ipv6-regexp () "Regexp matching left hand side of IPv6 addresses. Derived from `tramp-prefix-ipv6-format'." - (regexp-quote (tramp-prefix-ipv6-format))) + (tramp-lookup-syntax tramp-prefix-ipv6-regexp-alist)) ;; The following regexp is a bit sloppy. But it shall serve our ;; purposes. It covers also IPv4 mapped IPv6 addresses, like in @@ -789,18 +828,28 @@ tramp-ipv6-regexp "\\(?:\\(?:[a-zA-Z0-9]+\\)?:\\)+[a-zA-Z0-9.]+" "Regexp matching IPv6 addresses.") +(defconst tramp-postfix-ipv6-format-alist + '((default . "]") + (simplified . "]") + (separate . "")) + "Alist mapping Tramp syntax to suffix for IPv6 addresses.") + (defun tramp-postfix-ipv6-format () "String matching right hand side of IPv6 addresses. Used in `tramp-make-tramp-file-name'." - (cond ((eq (tramp-compat-tramp-syntax) 'default) "]") - ((eq (tramp-compat-tramp-syntax) 'simplified) "]") - ((eq (tramp-compat-tramp-syntax) 'separate) "") - (t (error "Wrong `tramp-syntax' %s" tramp-syntax)))) + (tramp-lookup-syntax tramp-postfix-ipv6-format-alist)) + +(defconst tramp-postfix-ipv6-regexp-alist + `((default . ,(regexp-quote "]")) + (simplified . ,(regexp-quote "]")) + (separate . ,(regexp-quote ""))) + "Alist mapping Tramp syntax to regexps matching IPv6 suffixes. +Derived from `tramp-postfix-ipv6-format-alist'.") (defun tramp-postfix-ipv6-regexp () "Regexp matching right hand side of IPv6 addresses. Derived from `tramp-postfix-ipv6-format'." - (regexp-quote (tramp-postfix-ipv6-format))) + (tramp-lookup-syntax tramp-postfix-ipv6-format-alist)) (defconst tramp-prefix-port-format "#" "String matching delimiter between host names and port numbers.") @@ -827,18 +876,28 @@ tramp-postfix-hop-regexp "Regexp matching delimiter after ad-hoc hop definitions. Derived from `tramp-postfix-hop-format'.") +(defconst tramp-postfix-host-format-alist + '((default . ":") + (simplified . ":") + (separate . "]")) + "Alist mapping Tramp syntax to strings between host and local names.") + (defun tramp-postfix-host-format () "String matching delimiter between host names and localnames. Used in `tramp-make-tramp-file-name'." - (cond ((eq (tramp-compat-tramp-syntax) 'default) ":") - ((eq (tramp-compat-tramp-syntax) 'simplified) ":") - ((eq (tramp-compat-tramp-syntax) 'separate) "]") - (t (error "Wrong `tramp-syntax' %s" tramp-syntax)))) + (tramp-lookup-syntax tramp-postfix-host-format-alist)) + +(defconst tramp-postfix-host-regexp-alist + `((default . ,(regexp-quote ":")) + (simplified . ,(regexp-quote ":")) + (separate . ,(regexp-quote "]"))) + "Alist mapping Tramp syntax to regexp matching name delimiters. +Derived from `tramp-postfix-host-format-alist'.") (defun tramp-postfix-host-regexp () "Regexp matching delimiter between host names and localnames. Derived from `tramp-postfix-host-format'." - (regexp-quote (tramp-postfix-host-format))) + (tramp-lookup-syntax tramp-postfix-host-regexp-alist)) (defconst tramp-localname-regexp ".*$" "Regexp matching localnames.") @@ -851,16 +910,46 @@ tramp-unknown-id-integer ;;; File name format: +(defun tramp-build-remote-file-name-spec-regexp (syntax) + "Construct a regexp matching a Tramp file name for a Tramp SYNTAX." + (let ((tramp-syntax syntax)) + (concat + "\\(" (tramp-method-regexp) "\\)" (tramp-postfix-method-regexp) + "\\(?:" "\\(" tramp-user-regexp "\\)" tramp-postfix-user-regexp "\\)?" + "\\(" "\\(?:" tramp-host-regexp "\\|" + (tramp-prefix-ipv6-regexp) + "\\(?:" tramp-ipv6-regexp "\\)?" + (tramp-postfix-ipv6-regexp) "\\)?" + "\\(?:" tramp-prefix-port-regexp tramp-port-regexp "\\)?" "\\)?"))) + +(defconst tramp-remote-file-name-spec-regexp-alist + `((default . ,(tramp-build-remote-file-name-spec-regexp 'default)) + (simplified . ,(tramp-build-remote-file-name-spec-regexp 'simplified)) + (separate . ,(tramp-build-remote-file-name-spec-regexp 'separate))) + "Alist mapping Tramp syntax to regexps matching Tramp file names.") + (defun tramp-remote-file-name-spec-regexp () "Regular expression matching a Tramp file name between prefix and postfix." - (concat - "\\(" (tramp-method-regexp) "\\)" (tramp-postfix-method-regexp) - "\\(?:" "\\(" tramp-user-regexp "\\)" tramp-postfix-user-regexp "\\)?" - "\\(" "\\(?:" tramp-host-regexp "\\|" - (tramp-prefix-ipv6-regexp) - "\\(?:" tramp-ipv6-regexp "\\)?" - (tramp-postfix-ipv6-regexp) "\\)?" - "\\(?:" tramp-prefix-port-regexp tramp-port-regexp "\\)?" "\\)?")) + (tramp-lookup-syntax tramp-remote-file-name-spec-regexp-alist)) + +(defun tramp-build-file-name-structure (syntax) + "Construct the Tramp file name structure for SYNTAX. +See `tramp-file-name-structure'." + (let ((tramp-syntax syntax)) + (list + (concat + (tramp-prefix-regexp) + "\\(" "\\(?:" (tramp-remote-file-name-spec-regexp) + tramp-postfix-hop-regexp "\\)+" "\\)?" + (tramp-remote-file-name-spec-regexp) (tramp-postfix-host-regexp) + "\\(" tramp-localname-regexp "\\)") + 5 6 7 8 1))) + +(defconst tramp-file-name-structure-alist + `((default . ,(tramp-build-file-name-structure 'default)) + (simplified . ,(tramp-build-file-name-structure 'simplified)) + (separate . ,(tramp-build-file-name-structure 'separate))) + "Alist mapping Tramp syntax to the file name structure for that syntax.") (defun tramp-file-name-structure () "List of six elements (REGEXP METHOD USER HOST FILE HOP), detailing \ @@ -881,14 +970,7 @@ tramp-file-name-structure means the opening parentheses are counted to identify the pair. See also `tramp-file-name-regexp'." - (list - (concat - (tramp-prefix-regexp) - "\\(" "\\(?:" (tramp-remote-file-name-spec-regexp) - tramp-postfix-hop-regexp "\\)+" "\\)?" - (tramp-remote-file-name-spec-regexp) (tramp-postfix-host-regexp) - "\\(" tramp-localname-regexp "\\)") - 5 6 7 8 1)) + (tramp-lookup-syntax tramp-file-name-structure-alist)) (defun tramp-file-name-regexp () "Regular expression matching file names handled by Tramp. -- 2.12.2 --=-=-= Content-Type: text/plain npostavs@users.sourceforge.net writes: > Gemini Lasswell writes: > >> +(defmacro tramp-lookup-syntax (alist) >> + "Look up a syntax string in ALIST according to `tramp-compat-tramp-syntax.' >> +Raise an error if `tramp-syntax' is invalid." >> + `(let ((result (cdr (assq (tramp-compat-tramp-syntax) ,alist)))) >> + (or result >> + (error "Wrong `tramp-syntax' %s" tramp-syntax)))) > > Was there a reason to defmacro instead of defun here? If it's > performance, perhaps defsubst could work instead? > > (also, the let-binding seems redundant) --=-=-=--