From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Spencer Baugh Newsgroups: gmane.emacs.bugs Subject: bug#71094: [PATCH] Prefer to run find and grep in parallel in rgrep Date: Wed, 22 May 2024 08:54:25 -0400 Message-ID: References: <86ttiq6or8.fsf@gnu.org> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="19844"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Gnus/5.13 (Gnus v5.13) Cc: rgm@gnu.org, 71094@debbugs.gnu.org, dmitry@gutov.dev To: Eli Zaretskii Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Wed May 22 14:55:37 2024 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1s9lVI-0004sl-EZ for geb-bug-gnu-emacs@m.gmane-mx.org; Wed, 22 May 2024 14:55:36 +0200 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1s9lUg-0003zC-6c; Wed, 22 May 2024 08:54:58 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1s9lUe-0003yW-Ma for bug-gnu-emacs@gnu.org; Wed, 22 May 2024 08:54:56 -0400 Original-Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1s9lUe-0001rY-AI for bug-gnu-emacs@gnu.org; Wed, 22 May 2024 08:54:56 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1s9lUk-0002PK-1m for bug-gnu-emacs@gnu.org; Wed, 22 May 2024 08:55:02 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Spencer Baugh Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Wed, 22 May 2024 12:55:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 71094 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: patch Original-Received: via spool by 71094-submit@debbugs.gnu.org id=B71094.17163824799235 (code B ref 71094); Wed, 22 May 2024 12:55:02 +0000 Original-Received: (at 71094) by debbugs.gnu.org; 22 May 2024 12:54:39 +0000 Original-Received: from localhost ([127.0.0.1]:55789 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s9lUM-0002Ot-Rw for submit@debbugs.gnu.org; Wed, 22 May 2024 08:54:39 -0400 Original-Received: from mxout6.mail.janestreet.com ([64.215.233.21]:59437) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s9lUK-0002On-Jn for 71094@debbugs.gnu.org; Wed, 22 May 2024 08:54:37 -0400 In-Reply-To: <86ttiq6or8.fsf@gnu.org> (Eli Zaretskii's message of "Wed, 22 May 2024 14:59:39 +0300") DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=janestreet.com; s=waixah; t=1716382465; bh=McatyCaRI/rnJuGuONJuYNotmBaiGyUAowPHXu13Ti8=; h=From:To:Cc:Subject:In-Reply-To:References:Date; b=rEDEK9u2TdoAnzJ2tZb1nh+M6lMRV36ka9q3SCEPZawJuoUMZ/jn3NJpZgAhj2L5t ebqnG4BQurcOD0Bz0cIYjMUV7bAEC6rot6+eLg4o8zrOY7+KKzc+3Fe1nU6PMjIUH7 zODfdKcfWL9eOPFxK8FFDcHEFw1XUIOsBZ42g8HWPP1CnN9fJx4LW/i3qaYxD3uU29 NlMeHH/c2xpo5aW9AzaaEhsENxZHntO0uWeyijudI2lHTM+qP5y5k/HUYYzVPfwMqp fBqBPqSsmmgfPehiljZmeXIxw0UJMT/MT4Wh+NR/PyAQxWcW1SeGOh4idICHOuXV78 PctE1DDET0z2w== X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.bugs:285610 Archived-At: --=-=-= Content-Type: text/plain Eli Zaretskii writes: >> Cc: Glenn Morris , dmitry@gutov.dev >> From: Spencer Baugh >> Date: Tue, 21 May 2024 10:35:07 -0400 >> >> grep.el prefers to run "find" and "xargs grep" in a pipeline, >> which means that "find" can continue searching the filesystem >> while "xargs grep" searches files. If find and xargs don't >> support the flags required for this behavior, grep.el will fall >> back to using the -exec flags to "find", which meant "find" will >> wait for each "grep" process to complete before continuing to >> search the filesystem tree. This behavior is controlled by >> grep-find-use-xargs; `gnu' produces the pipeline and `exec' is >> the slower fallback. >> >> In f3ca7378c1336b3ff98ecb5a99a98c7b2eceece9, the `exec-plus' >> option was added for grep-find-use-xargs, which improves on >> `exec' by running one "grep" process to search multiple files, >> which `gnu' (by using xargs) already did. However, the change >> erroneously added the `exec-plus' case before the `gnu' case in >> the autodetection code in grep-compute-defaults, so `exec-plus' >> would be used even if `gnu' was supported. >> >> This change just swaps the two cases, so the faster `gnu' option >> is once again used in preference to `exec-plus'. In my >> benchmarking on a large repository, this provides a ~40% >> speedup. > > With how many files did you measure the 40% speedup? 700k > Can you show the performance with much fewer and much more files than > what you used? Much more is maybe hard, but much fewer is easy: with 212 files (a subset of the original directory I searched), there's no performance change. > I suspect that the effect depends on that. (It also depends on the >system limit on the number of files and the length of the command line >that xargs can use.) The argument about 'find' waiting is no longer >relevant with 'exec-plus', since in most cases there will be just one >invocation of 'grep'. True, it only matters when the directory tree contains more files than can be passed to a single invocation of grep. > In any case, please modify the patch so that 'exec-plus' is still > preferred on MS-Windows (because most Windows ports of xargs are IME > abysmally buggy, so better avoided as much as possible). > > A comment there with the justification of the order will also be > appreciated. Done, attached. --=-=-= Content-Type: text/x-patch Content-Disposition: inline; filename=0001-Prefer-to-run-find-and-grep-in-parallel-in-rgrep.patch >From e7fbfe431ae1f4f004f1d92db2f3b011b30ff682 Mon Sep 17 00:00:00 2001 From: Spencer Baugh Date: Tue, 21 May 2024 10:32:45 -0400 Subject: [PATCH] Prefer to run find and grep in parallel in rgrep grep.el prefers to run "find" and "xargs grep" in a pipeline, which means that "find" can continue searching the filesystem while "xargs grep" searches files. If find and xargs don't support the flags required for this behavior, grep.el will fall back to using the -exec flags to "find", which meant "find" will wait for each "grep" process to complete before continuing to search the filesystem tree. This behavior is controlled by grep-find-use-xargs; `gnu' produces the pipeline and `exec' is the slower fallback. In f3ca7378c1336b3ff98ecb5a99a98c7b2eceece9, the `exec-plus' option was added for grep-find-use-xargs, which improves on `exec' by running one "grep" process to search multiple files, which `gnu' (by using xargs) already did. However, the change erroneously added the `exec-plus' case before the `gnu' case in the autodetection code in grep-compute-defaults, so `exec-plus' would be used even if `gnu' was supported. This change just swaps the two cases, so the faster `gnu' option is once again used in preference to `exec-plus'. In my benchmarking on a large repository, this provides a ~40% speedup. Also, we completely avoid running xargs on MS-Windows, because Eli Zaretskii writes: > most Windows ports of xargs are IME abysmally buggy, so better avoided > as much as possible * lisp/progmodes/grep.el (grep-compute-defaults): Prefer `gnu' for grep-find-use-xargs over `exec-plus', but not on Windows. (bug#71094) --- lisp/progmodes/grep.el | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/lisp/progmodes/grep.el b/lisp/progmodes/grep.el index 0a9de04fce1..ce54c57aabc 100644 --- a/lisp/progmodes/grep.el +++ b/lisp/progmodes/grep.el @@ -812,15 +812,23 @@ grep-compute-defaults (unless grep-find-use-xargs (setq grep-find-use-xargs (cond - ((grep-probe find-program - `(nil nil nil ,(null-device) "-exec" "echo" - "{}" "+")) - 'exec-plus) + ;; For performance, we want: + ;; A. Run grep on batches of files (instead of one grep per file) + ;; B. If the directory is large and we need multiple batches, + ;; run find in parallel with a running grep. + ;; "find | xargs grep" gives both A and B ((and + (not (eq system-type 'windows-nt)) (grep-probe find-program `(nil nil nil ,(null-device) "-print0")) (grep-probe xargs-program '(nil nil nil "-0" "echo"))) 'gnu) + ;; "find -exec {} +" gives A but not B + ((grep-probe find-program + `(nil nil nil ,(null-device) "-exec" "echo" + "{}" "+")) + 'exec-plus) + ;; "find -exec {} ;" gives neither A nor B. (t 'exec)))) (unless grep-find-command -- 2.39.3 --=-=-=--