bug#36857: 27.0.50; Reduce calls to xref-collect-matches in dired-do-find-regexp

bug#36857: 27.0.50; Reduce calls to xref-collect-matches in dired-do-find-regexp

[Eric Abrahamsen]

This is a proper bug report for an issue I raised on emacs.devel:

When marking multiple files in Dired, then running a search with
`dired-do-find-regexp', `xref-collect-matches' is called once per marked
file, when it could just be called once for the whole search. This can
slow things down quite a bit.

bug#36857: 27.0.50; Reduce calls to xref-collect-matches in dired-do-find-regexp

[Dmitry Gutov]

On 30.07.2019 20:20, Eric Abrahamsen wrote:
> When marking multiple files in Dired

Could you estimate how many files you need to mark for this to become a 
problem? I'm really curious.

bug#36857: 27.0.50; Reduce calls to xref-collect-matches in dired-do-find-regexp

[Dmitry Gutov]

On 30.07.2019 21:44, Dmitry Gutov wrote:
> On 30.07.2019 20:20, Eric Abrahamsen wrote:
>> When marking multiple files in Dired
> 
> Could you estimate how many files you need to mark for this to become a 
> problem? I'm really curious.

Never mind, saw your reply on emacs-devel. 200-300 is a respectable amount.

bug#36857: 27.0.50; Reduce calls to xref-collect-matches in dired-do-find-regexp

[Eric Abrahamsen]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 30.07.2019 21:44, Dmitry Gutov wrote:
>> On 30.07.2019 20:20, Eric Abrahamsen wrote:
>>> When marking multiple files in Dired
>>
>> Could you estimate how many files you need to mark for this to
>> become a problem? I'm really curious.
>
> Never mind, saw your reply on emacs-devel. 200-300 is a respectable amount.

Does "respectable" mean "quite a few" or "a number we should be able to
handle without significant slowdowns"? :)

bug#36857: 27.0.50; Reduce calls to xref-collect-matches in dired-do-find-regexp

[Dmitry Gutov]

On 30.07.2019 21:56, Eric Abrahamsen wrote:

>> Never mind, saw your reply on emacs-devel. 200-300 is a respectable amount.
> 
> Does "respectable" mean "quite a few" or "a number we should be able to
> handle without significant slowdowns"? :)

Hopefully both. :-)

I have an improvement in mind that would fix it, but there are a couple 
change I'll need to make in a related area (project.el) first, so please 
stay tuned.

bug#36857: 27.0.50; Reduce calls to xref-collect-matches in dired-do-find-regexp

[Eric Abrahamsen]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 30.07.2019 21:56, Eric Abrahamsen wrote:
>
>>> Never mind, saw your reply on emacs-devel. 200-300 is a respectable amount.
>>
>> Does "respectable" mean "quite a few" or "a number we should be able to
>> handle without significant slowdowns"? :)
>
> Hopefully both. :-)
>
> I have an improvement in mind that would fix it, but there are a
> couple change I'll need to make in a related area (project.el) first,
> so please stay tuned.

Cool, thanks!

bug#36857: 27.0.50; Reduce calls to xref-collect-matches in dired-do-find-regexp

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 95 bytes --]

Hi Eric,

Please try the attached patch. It should cut down on the number of 
process calls.



[-- Attachment #2: dired-do-find-regexp-speedup.diff --]
[-- Type: text/x-patch, Size: 1585 bytes --]

diff --git a/lisp/dired-aux.el b/lisp/dired-aux.el
index fb1ad6266d..0fb8839a3f 100644
--- a/lisp/dired-aux.el
+++ b/lisp/dired-aux.el
@@ -2975,19 +2975,24 @@ dired-do-find-regexp
   (require 'xref)
   (defvar grep-find-ignored-files)
   (declare-function rgrep-find-ignored-directories "grep" (dir))
-  (let* ((files (dired-get-marked-files nil nil nil nil t))
+  (let* ((marks (dired-get-marked-files nil nil nil nil t))
          (ignores (nconc (mapcar
                           #'file-name-as-directory
                           (rgrep-find-ignored-directories default-directory))
                          grep-find-ignored-files))
          (fetcher
           (lambda ()
-            (let ((xrefs (mapcan
-                          (lambda (file)
-                            (xref-collect-matches regexp "*" file
-                                                  (and (file-directory-p file)
-                                                       ignores)))
-                          files)))
+            (let (files xrefs)
+              (mapc
+               (lambda (mark)
+                 (if (file-directory-p mark)
+                     (setq files (nconc
+                                  (project--files-in-directory mark ignores "*")
+                                  files))
+                   (push mark files)))
+               (nreverse marks))
+              (setq xrefs
+                    (project--find-regexp-in-files regexp files))
               (unless xrefs
                 (user-error "No matches for: %s" regexp))
               xrefs))))

bug#36857: 27.0.50; Reduce calls to xref-collect-matches in dired-do-find-regexp

[Eric Abrahamsen]

On 12/25/19 18:31 PM, Dmitry Gutov wrote:
> Hi Eric,
>
> Please try the attached patch. It should cut down on the number of
> process calls.

That's great! The searches that once took "longer than I was willing to
wait" now return nearly instantly. Thanks a lot. The compiler complains
that `project--files-in-directory' and `project--find-regexp-in-files'
are unknown, but presumably that's easily addressed.

Thanks,
Eric

bug#36857: 27.0.50; Reduce calls to xref-collect-matches in dired-do-find-regexp

[Dmitry Gutov]

On 25.12.2019 23:46, Eric Abrahamsen wrote:

>> Please try the attached patch. It should cut down on the number of
>> process calls.
> 
> That's great! The searches that once took "longer than I was willing to
> wait" now return nearly instantly. Thanks a lot. The compiler complains
> that `project--files-in-directory' and `project--find-regexp-in-files'
> are unknown, but presumably that's easily addressed.

Thanks for testing!

Pushed as ccd7cd2c51.