Eli Zaretskii <eliz@gnu.org> writes:
>> From: Spencer Baugh <sbaugh@janestreet.com>
>> Date: Wed, 13 Mar 2024 11:01:05 -0400
>> 
>> In my benchmarking, for large dired buffers, using regexp-opt provides
>> around a 3x speedup in omitting.
>
> Can you show a recipe for such benchmarking?  I'd like to try that on
> my systems.
>
> Also, what is the slowdown in the (improbable, but possible) case
> where dired-omit-extensions change for each call of dired-omit-regexp?

Yes, run the following after applying the patch:

(require 'dired)
(require 'dired-x)
(require 'cl-lib)

(defun dired-omit-regexp-old ()
  (concat (if dired-omit-files (concat "\\(" dired-omit-files "\\)") "")
          (if (and dired-omit-files dired-omit-extensions) "\\|" "")
          (if dired-omit-extensions
              (concat ".";; a non-extension part should exist
                      "\\("
                      (mapconcat 'regexp-quote dired-omit-extensions "\\|")
                      "\\)$")
            "")))

(defun my-do-omit (mode)
  (let ((regexp
	 (cl-case mode
	   (new (dired-omit-regexp))
	   (old (dired-omit-regexp-old))
	   (new-uncached (let ((dired-omit--extension-regexp-cache nil)) (dired-omit-regexp)))
	   (t (error "Bad mode %s" mode)))))
    (dired-mark-if
     (let ((fn (dired-get-filename nil t)))
       (and fn (string-match-p regexp fn)))
     nil)))

(defun my-bench-omit (nfiles ntimes)
  (let ((default-directory (expand-file-name "test-dired-list")))
    (make-directory default-directory t)
    (dolist (file (directory-files "." t "test-file"))
      (delete-file file))
    (dotimes (i nfiles)
      (write-region "" nil (format "test-file%s" i) nil 'nomessage nil 'excl))
    (let ((dired-omit-mode nil))
      (with-current-buffer (let ((inhibit-message t)) (dired-noselect default-directory))
	(revert-buffer)
	(message "files %s, ntimes %s: new %s old %s new-uncached %s"
		 nfiles ntimes
		 (car (benchmark-call (lambda () (my-do-omit 'new)) ntimes))
		 (car (benchmark-call (lambda () (my-do-omit 'old)) ntimes))
		 (car (benchmark-call (lambda () (my-do-omit 'new-uncached)) ntimes)))
	))))

(my-bench-omit 1 100)
(my-bench-omit 10 100)
(my-bench-omit 100 100)
(my-bench-omit 1000 100)
(my-bench-omit 10000 100)

For me, I get:
$ ./src/emacs -Q --batch -l ../emacs-29/bench-omit.elc
files 1, ntimes 100: new 0.008839979999999999 old 0.018162129 new-uncached 0.031399762
files 10, ntimes 100: new 0.012037615 old 0.040232355000000004 new-uncached 0.037990543
files 100, ntimes 100: new 0.07368538100000001 old 0.314905271 new-uncached 0.10006527300000001
files 1000, ntimes 100: new 0.669103498 old 3.076339984 new-uncached 0.693134644
files 10000, ntimes 100: new 6.336211434 old 30.926320486 new-uncached 6.442762152999999

So the performance improvement is quite substantial for large
directories.

new-uncached is the performance if dired-omit-extensions changes on each
call of dired-omit-regexp.  For a directory of 1 file, the overhead of
recomputing regexp-opt every time makes the performance perhaps 2x-3x
worse, but around 10 files the performance improvement from regexp-opt
exceeds the overhead, and above that the uncached version still
outperforms the old version substantially.

If dired-omit-extensions doesn't change every time, the performance is
improved even for directories of 1 file.

>> regexp-opt takes around 5 milliseconds, so to avoid slowing down
>> omitting in small dired buffers we cache the return value.
>> 
>> Since omitting is now 3x faster, increase dired-omit-size-limit by 3x.
>> 
>> * lisp/dired-x.el (dired-omit--extension-regexp-cache): Add.
>> (dired-omit-regexp): Use regexp-opt.
>> (dired-omit-size-limit): Increase, since omitting is now faster.
>
> I'm okay with these changes, but:
>
>   . the change in the default value of dired-omit-size-limit should be
>     called out in NEWS
>   . please document this variable in the dired-x.texi manual, where we
>     document all the other variables relevant to dired-omit mode.
>   . the doc string of dired-omit-size-limit is embarrassingly
>     unhelpful, so bonus points for fixing that as well
>
> Thanks.

Certainly, updated patch attached.