From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED.blaine.gmane.org!not-for-mail From: Michael Albinus Newsgroups: gmane.emacs.devel Subject: Re: TRAMP problem with large repositories Date: Thu, 12 Dec 2019 14:35:50 +0100 Message-ID: <877e313bux.fsf@gmx.de> References: Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Injection-Info: blaine.gmane.org; posting-host="blaine.gmane.org:195.159.176.226"; logging-data="68611"; mail-complaints-to="usenet@blaine.gmane.org" User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux) Cc: Emacs developers To: Philippe Vaucher Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Thu Dec 12 15:10:26 2019 Return-path: Envelope-to: ged-emacs-devel@m.gmane.org Original-Received: from lists.gnu.org ([209.51.188.17]) by blaine.gmane.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from ) id 1ifPAf-000Hhs-L9 for ged-emacs-devel@m.gmane.org; Thu, 12 Dec 2019 15:10:25 +0100 Original-Received: from localhost ([::1]:59988 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ifPAe-0006Bi-0b for ged-emacs-devel@m.gmane.org; Thu, 12 Dec 2019 09:10:24 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:46909) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ifOdK-0006wU-8L for emacs-devel@gnu.org; Thu, 12 Dec 2019 08:35:59 -0500 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ifOdI-0006pe-G2 for emacs-devel@gnu.org; Thu, 12 Dec 2019 08:35:58 -0500 Original-Received: from mout.gmx.net ([212.227.17.20]:46609) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1ifOdI-0006kD-0o for emacs-devel@gnu.org; Thu, 12 Dec 2019 08:35:56 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net; s=badeba3b8450; t=1576157752; bh=ipmbMgS1A9nUY95s4zt4UuK85XfA14Ev4Wce8+q8auk=; h=X-UI-Sender-Class:From:To:Cc:Subject:References:Date; b=awog6PYUuMBPsbOgU7yKvCGmSOPve3Q7pAjmLcMlLhYoCrYQD8GLIFSQVcl+rziWX XBu8VdHWCbjB+w0YLLOAe8V6JNCp02rbSr7SntUrZnQzdqcQEpD/fyqcjlY5pZaB/h Zu20yI48jXIcam3Dg6EhQi3VB0QiVPirg0Ryri+I= X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c Original-Received: from detlef.gmx.de ([212.86.57.48]) by mail.gmx.com (mrgmx104 [212.227.17.168]) with ESMTPSA (Nemesis) id 1MVeI8-1iG4H534QW-00RcS6; Thu, 12 Dec 2019 14:35:51 +0100 X-Provags-ID: V03:K1:el9xDVTlI4MaKMnFycX/ukD0a5Y2WF/+ZB5cR4+E3XpFMeOiqFA toLVldSr1+bESkO2144OC0ljSsZF4XERnG8GP8afiu9LT3LR8U3eQmPv9Bj7ouVOrzavL5h mZimoiutILQM4X+RDcKL5l9bERAf2x3twe5f+AruiZxukmYpkjF+kK8pSsNeQq/kevP1I8D H1McsCw+RcxVSadlx1Q+w== X-UI-Out-Filterresults: notjunk:1;V03:K0:QhEoNLykNCI=:x70SGh1PseMPaXffbSVTXt aoOX26paowe7jzqWyWWtxdUSBx3y2LkVJEQTb5vzdwLTVgy0J1yEWGLNv7szHvgOAC1dDQJJK mlmgSbArBtCifUurOzAIpGD4LZLkdEtLW0NO3NPd+NsipMiRdyFIFgeihTpcEAnfBxboNYysB tTYIluGhnkicT+ONgGdVtEjuwLTdHsQXAidKmZhy0Gu2aYMcTK8kSNAbxGWVmP7Fr6+TSzchC jbTHZ5lyfSN9zOU9BlWnbYYdk2A/iF4lNW26+ClVh3Sugki+OBJDDNWu+zExEih68U55Z8Fi4 D2qtrLNUGLPG87mtkQmvopFZ/MnI9Y0JDtiacDgWs8EIPw9jYUnTUhTCJ5r1QJ+JBDYXFF/FZ dLIOLBQbU6dsws3Vci3jNdZzLNuES9mzAlVXOi3GjQL27LmGGuz1RzkgYbpFII/07tzHXWneU 3lxgpu8I+5Dhcc+iRTg7CSAeyy2CinAsBNqi4bNPLokponW4+R0/tc11/M3Dwl4/O4KNGUt/C AGUTQhQnVA08SAZ2cabaKzsN+vUl6Je84+OKLTjAfQhNBn+VIMaNN2Sx+5sipM5BEgMMtW16p EiMTsb6+GtORvp7+ZeLY1Zp6EmyuqXHu/+GATKtmj71Jzs2tnNZZ1iYIEko2951rZbIr7+Cv9 0+tv7PIMKHi7PUqJ3M3oYyF6zZRO5BSf/Ud/lJ8GS0AZXX5jOjGxXpvdaA4wLPA2s0BCEYNO+ 7J5C1w/ubix4FfSOIAanGu8Tykqa4fVkjepH8+amgXlR0WX04rPqvWWa6D1BfnF7WlAgrfik X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 212.227.17.20 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane.org@gnu.org Original-Sender: "Emacs-devel" Xref: news.gmane.org gmane.emacs.devel:243316 Archived-At: --=-=-= Content-Type: text/plain Philippe Vaucher writes: > Hello, Hi Philippe, > While helping someone for a projectile issue > (https://github.com/bbatsov/projectile/issues/1480), it seems that > when `shell-command-to-string` tries to execute `git ls-files -zco - > -exclude-standard` over TRAMP on a repository that has 85K files it > takes forever to complete. > > We see that `tramp-wait-for-output` calls `tramp-wait-for-regexp` > which calls `tramp-check-for-regexp`, and when looking at the source: > > My understanding is that it does a loop that reads a bit of what the > commands outputs then tries to parse end of lines (or '\0') and > repeats until the process died or that it found one. Because the > command returns a huge string (85K files), this process of > read-regexp-repeat takes all the CPU (compared to reading the whole > chunk in one go and then trying to check for the regexp). > > My questions are the following: > > 1 Did I understand the problem right? Is this something known? Yes, your analysis is right. And no, I haven't seen related reports yet. > 2 Is there something to be done about this? Or maybe it would it > require too much refactoring / faster implementation? I have appended a patch which should fix the problem. Could you, please, (let) test? Btw, the latest Tramp release is always available via GNU ELPA. > Kind regards, > Philippe Best regards, Michael. --=-=-= Content-Type: text/x-patch Content-Disposition: attachment Content-Transfer-Encoding: quoted-printable diff --git a/lisp/tramp-sh.el b/lisp/tramp-sh.el index 8de88d35..506c33df 100644 =2D-- a/lisp/tramp-sh.el +++ b/lisp/tramp-sh.el @@ -5102,9 +5102,8 @@ function waits for output unless NOOUTPUT is set." (forward-line 1) (delete-region (point-min) (point))) ;; Delete the prompt. - (goto-char (point-max)) - (re-search-backward regexp nil t) - (delete-region (point) (point-max))) + (when (tramp-search-regexp regexp) + (delete-region (point) (point-max)))) (if timeout (tramp-error proc 'file-error @@ -5134,8 +5133,7 @@ DONT-SUPPRESS-ERR is non-nil, stderr won't be sent t= o /dev/null." "echo tramp_exit_status $?" (if subshell " )" ""))) (with-current-buffer (tramp-get-connection-buffer vec) - (goto-char (point-max)) - (unless (re-search-backward "tramp_exit_status [0-9]+" nil t) + (unless (tramp-search-regexp "tramp_exit_status [0-9]+") (tramp-error vec 'file-error "Couldn't find exit status of `%s'" command)) (skip-chars-forward "^ ") diff --git a/lisp/tramp.el b/lisp/tramp.el index 03e04568..e05e0965 100644 =2D-- a/lisp/tramp.el +++ b/lisp/tramp.el @@ -4196,19 +4196,35 @@ for process communication also." (buffer-string)) result))) +(defun tramp-search-regexp (regexp) + "Search for REGEXP backwards, starting at point-max. +If found, set point to the end of the occurrence found, and return point. +Otherwise, return nil." + (goto-char (point-max)) + ;; We restrict ourselves to the last 256 characters. There were + ;; reports of 85kB output, which has blocked Tramp forever. + (re-search-backward regexp (max (point-min) (- (point) 256)) 'noerror)) + (defun tramp-check-for-regexp (proc regexp) "Check, whether REGEXP is contained in process buffer of PROC. Erase echoed commands if exists." (with-current-buffer (process-buffer proc) (goto-char (point-min)) - ;; Check whether we need to remove echo output. + ;; Check whether we need to remove echo output. The max length of + ;; the echo mark regexp is taken for search. We restrict the + ;; search for the second echo mark to PIPE_BUF characters. (when (and (tramp-get-connection-property proc "check-remote-echo" ni= l) - (re-search-forward tramp-echoed-echo-mark-regexp nil t)) + (re-search-forward + tramp-echoed-echo-mark-regexp + (+ (point) (* 5 tramp-echo-mark-marker-length)) t)) (let ((begin (match-beginning 0))) - (when (re-search-forward tramp-echoed-echo-mark-regexp nil t) + (when + (re-search-forward + tramp-echoed-echo-mark-regexp + (+ (point) (tramp-get-connection-property proc "pipe-buf" 4096)) t) ;; Discard echo from remote output. - (tramp-set-connection-property proc "check-remote-echo" nil) + (tramp-flush-connection-property proc "check-remote-echo") (tramp-message proc 5 "echo-mark found") (forward-line 1) (delete-region begin (point)) @@ -4229,8 +4245,7 @@ Erase echoed commands if exists." ;; overflow in regexp matcher". For example, //DIRED// lines of ;; directory listings with some thousand files. Therefore, we ;; look from the end. - (goto-char (point-max)) - (ignore-errors (re-search-backward regexp nil t))))) + (tramp-search-regexp regexp)))) (defun tramp-wait-for-regexp (proc timeout regexp) "Wait for a REGEXP to appear from process PROC within TIMEOUT seconds. --=-=-=--