bug#64735: 29.0.92; find invocations are ~15x slower because of ignores

[Ihor Radchenko <yantar92@posteo.net>]

Dmitry Gutov <dmitry@gutov.dev> writes:

>>> ... Last I checked, Lisp-native file
>>> listing was simply slower than 'find'.
>> 
>> Could it be changed?
>> In my tests, I was able to improve performance of the built-in
>> `directory-files-recursively' simply by disabling
>> `file-name-handler-alist' around its call.
>
> Then it won't work with Tramp, right? I think it's pretty nifty that 
> project-find-regexp and dired-do-find-regexp work over Tramp.

Sure. It might also be optimized. Without trying to convince find devs
to do something about regexp handling.

And things are not as horrible as 15x slowdown in find.

>> See https://yhetil.org/emacs-devel/87cz0p2xlc.fsf@localhost/
>> (the thread also continues off-list, and it looks like there is a lot of
>> room for improvement in this area)
>
> Does it get close enough to the performance of 'find' this way?

Comparable:

(ignore (let ((gc-cons-threshold most-positive-fixnum)) (benchmark-progn (directory-files-recursively "/home/yantar92/.data" ""))))
;; Elapsed time: 0.633713s
(ignore (let ((gc-cons-threshold most-positive-fixnum)) (benchmark-progn (let ((file-name-handler-alist)) (directory-files-recursively "/home/yantar92/.data" "")))))
;; Elapsed time: 0.324341s
;; time find /home/yantar92/.data >/dev/null
;; real	0m0.129s
;; user	0m0.017s
;; sys	0m0.111s

> Also note that processing all matches in Lisp, with many ignores 
> entries, will incur the proportional overhead in Lisp. Which might be 
> relatively slow as well.

Not significant.
I tried to unwrap recursion in `directory-files-recursively' and tried
to play around with regexp matching of the file list itself - no
significant impact compared to `file-name-handler-alist'.

I am pretty sure that Emacs's native file routines can be optimized to
the level of find.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>