From: Dmitry Gutov <dmitry@gutov.dev>
To: Eli Zaretskii <eliz@gnu.org>
Cc: luangruo@yahoo.com, sbaugh@janestreet.com, yantar92@posteo.net,
64735@debbugs.gnu.org
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Date: Sat, 29 Jul 2023 03:12:34 +0300 [thread overview]
Message-ID: <59c30342-a7e0-d83b-a128-0faae4cbd633@gutov.dev> (raw)
In-Reply-To: <35f8b664-0241-9f96-1aa0-20ca51b2d34c@gutov.dev>
On 27/07/2023 16:30, Dmitry Gutov wrote:
> I can imagine that the filter-based approach necessarily creates more
> strings (to pass to the filter function). Maybe we could increase those
> strings' size (thus reducing the number) by increasing the read buffer
> size?
To go further along this route, first of all, I verified that the input
strings are (almost) all the same length: 4096. And they are parsed into
strings with length 50-100 characters, meaning the number of "junk"
objects due to the process-filter approach probably shouldn't matter too
much, given that the number of strings returned is 40-80x more.
But then I ran these tests with different values of
read-process-output-max, which exactly increased those strings' size,
proportionally reducing their number. The results were:
> (my-bench-rpom 1 default-directory "")
=>
(("with-find-p 4096" . "Elapsed time: 0.945478s (0.474680s in 6 GCs)")
("with-find-p 40960" . "Elapsed time: 0.760727s (0.395379s in 5 GCs)")
("with-find-p 409600" . "Elapsed time: 0.729757s (0.394881s in 5 GCs)"))
where
(defun my-bench-rpom (count path regexp)
(setq path (expand-file-name path))
(list
(cons "with-find-p 4096"
(let ((read-process-output-max 4096))
(benchmark count (list 'find-directory-files-recursively-2
path regexp))))
(cons "with-find-p 40960"
(let ((read-process-output-max 40960))
(benchmark count (list 'find-directory-files-recursively-2
path regexp))))
(cons "with-find-p 409600"
(let ((read-process-output-max 409600))
(benchmark count (list 'find-directory-files-recursively-2
path regexp))))))
...with the last iteration showing consistently the same or better
performance than the "sync" version I benchmarked previously.
What does that mean for us? The number of strings in the heap is
reduced, but not by much (again, the result is a list with 43x more
elements). The combined memory taken up by these intermediate strings to
be garbage-collected, is the same.
It seems like per-chunk overhead is non-trivial, and affects GC somehow
(but not in a way that just any string would).
In this test, by default, the output produces ~6000 strings and passes
them to the filter function. Meaning, read_and_dispose_of_process_output
is called about 6000 times, producing the overhead of roughly 0.2s.
Something in there must be producing extra work for the GC.
This line seems suspect:
list3 (outstream, make_lisp_proc (p), text),
Creates 3 conses and one Lisp object (tagged pointer). But maybe I'm
missing something bigger.
next prev parent reply other threads:[~2023-07-29 0:12 UTC|newest]
Thread overview: 202+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-07-19 21:16 bug#64735: 29.0.92; find invocations are ~15x slower because of ignores Spencer Baugh
2023-07-20 5:00 ` Eli Zaretskii
2023-07-20 12:22 ` sbaugh
2023-07-20 12:42 ` Dmitry Gutov
2023-07-20 13:43 ` Spencer Baugh
2023-07-20 18:54 ` Dmitry Gutov
2023-07-20 12:38 ` Dmitry Gutov
2023-07-20 13:20 ` Ihor Radchenko
2023-07-20 15:19 ` Dmitry Gutov
2023-07-20 15:42 ` Ihor Radchenko
2023-07-20 15:57 ` Dmitry Gutov
2023-07-20 16:03 ` Ihor Radchenko
2023-07-20 18:56 ` Dmitry Gutov
2023-07-21 9:14 ` Ihor Radchenko
2023-07-20 16:33 ` Eli Zaretskii
2023-07-20 16:36 ` Ihor Radchenko
2023-07-20 16:45 ` Eli Zaretskii
2023-07-20 17:23 ` Ihor Radchenko
2023-07-20 18:24 ` Eli Zaretskii
2023-07-20 18:29 ` Ihor Radchenko
2023-07-20 18:43 ` Eli Zaretskii
2023-07-20 18:57 ` Ihor Radchenko
2023-07-21 12:37 ` Dmitry Gutov
2023-07-21 12:58 ` Ihor Radchenko
2023-07-21 13:00 ` Dmitry Gutov
2023-07-21 13:34 ` Ihor Radchenko
2023-07-21 13:36 ` Dmitry Gutov
2023-07-21 13:46 ` Ihor Radchenko
2023-07-21 15:41 ` Dmitry Gutov
2023-07-21 15:48 ` Ihor Radchenko
2023-07-21 19:53 ` Dmitry Gutov
2023-07-23 5:40 ` Ihor Radchenko
2023-07-23 11:50 ` Michael Albinus
2023-07-24 7:35 ` Ihor Radchenko
2023-07-24 7:59 ` Michael Albinus
2023-07-24 8:22 ` Ihor Radchenko
2023-07-24 9:31 ` Michael Albinus
2023-07-21 7:45 ` Michael Albinus
2023-07-21 10:46 ` Eli Zaretskii
2023-07-21 11:32 ` Michael Albinus
2023-07-21 11:51 ` Ihor Radchenko
2023-07-21 12:01 ` Michael Albinus
2023-07-21 12:20 ` Ihor Radchenko
2023-07-21 12:25 ` Ihor Radchenko
2023-07-21 12:46 ` Eli Zaretskii
2023-07-21 13:01 ` Michael Albinus
2023-07-21 13:23 ` Ihor Radchenko
2023-07-21 15:31 ` Michael Albinus
2023-07-21 15:38 ` Ihor Radchenko
2023-07-21 15:49 ` Michael Albinus
2023-07-21 15:55 ` Eli Zaretskii
2023-07-21 16:08 ` Michael Albinus
2023-07-21 16:15 ` Ihor Radchenko
2023-07-21 16:38 ` Eli Zaretskii
2023-07-21 16:43 ` Ihor Radchenko
2023-07-21 16:43 ` Michael Albinus
2023-07-21 17:45 ` Eli Zaretskii
2023-07-21 17:55 ` Michael Albinus
2023-07-21 18:38 ` Eli Zaretskii
2023-07-21 19:33 ` Spencer Baugh
2023-07-22 5:27 ` Eli Zaretskii
2023-07-22 10:38 ` sbaugh
2023-07-22 11:58 ` Eli Zaretskii
2023-07-22 14:14 ` Ihor Radchenko
2023-07-22 14:32 ` Eli Zaretskii
2023-07-22 15:07 ` Ihor Radchenko
2023-07-22 15:29 ` Eli Zaretskii
2023-07-23 7:52 ` Ihor Radchenko
2023-07-23 8:01 ` Eli Zaretskii
2023-07-23 8:11 ` Ihor Radchenko
2023-07-23 9:11 ` Eli Zaretskii
2023-07-23 9:34 ` Ihor Radchenko
2023-07-23 9:39 ` Eli Zaretskii
2023-07-23 9:42 ` Ihor Radchenko
2023-07-23 10:20 ` Eli Zaretskii
2023-07-23 11:43 ` Ihor Radchenko
2023-07-23 12:49 ` Eli Zaretskii
2023-07-23 12:57 ` Ihor Radchenko
2023-07-23 13:32 ` Eli Zaretskii
2023-07-23 13:56 ` Ihor Radchenko
2023-07-23 14:32 ` Eli Zaretskii
2023-07-22 17:18 ` sbaugh
2023-07-22 17:26 ` Ihor Radchenko
2023-07-22 17:46 ` Eli Zaretskii
2023-07-22 18:31 ` Eli Zaretskii
2023-07-22 19:06 ` Eli Zaretskii
2023-07-22 20:53 ` Spencer Baugh
2023-07-23 6:15 ` Eli Zaretskii
2023-07-23 7:48 ` Ihor Radchenko
2023-07-23 8:06 ` Eli Zaretskii
2023-07-23 8:16 ` Ihor Radchenko
2023-07-23 9:13 ` Eli Zaretskii
2023-07-23 9:16 ` Ihor Radchenko
2023-07-23 11:44 ` Michael Albinus
2023-07-23 2:59 ` Richard Stallman
2023-07-23 5:28 ` Eli Zaretskii
2023-07-22 8:17 ` Michael Albinus
2023-07-21 13:17 ` Ihor Radchenko
2023-07-21 12:27 ` Michael Albinus
2023-07-21 12:30 ` Ihor Radchenko
2023-07-21 13:04 ` Michael Albinus
2023-07-21 13:24 ` Ihor Radchenko
2023-07-21 15:36 ` Michael Albinus
2023-07-21 15:44 ` Ihor Radchenko
2023-07-21 12:39 ` Eli Zaretskii
2023-07-21 13:09 ` Michael Albinus
2023-07-21 12:38 ` Dmitry Gutov
2023-07-20 17:08 ` Spencer Baugh
2023-07-20 17:24 ` Eli Zaretskii
2023-07-22 6:35 ` Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors
2023-07-20 17:25 ` Ihor Radchenko
2023-07-21 19:31 ` Spencer Baugh
2023-07-21 19:37 ` Ihor Radchenko
2023-07-21 19:56 ` Dmitry Gutov
2023-07-21 20:11 ` Spencer Baugh
2023-07-22 6:39 ` Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors
2023-07-22 21:01 ` Dmitry Gutov
2023-07-23 5:11 ` Eli Zaretskii
2023-07-23 10:46 ` Dmitry Gutov
2023-07-23 11:18 ` Eli Zaretskii
2023-07-23 17:46 ` Dmitry Gutov
2023-07-23 17:56 ` Eli Zaretskii
2023-07-23 17:58 ` Dmitry Gutov
2023-07-23 18:21 ` Eli Zaretskii
2023-07-23 19:07 ` Dmitry Gutov
2023-07-23 19:27 ` Eli Zaretskii
2023-07-23 19:44 ` Dmitry Gutov
2023-07-23 19:27 ` Dmitry Gutov
2023-07-24 11:20 ` Eli Zaretskii
2023-07-24 12:55 ` Dmitry Gutov
2023-07-24 13:26 ` Eli Zaretskii
2023-07-25 2:41 ` Dmitry Gutov
2023-07-25 8:22 ` Ihor Radchenko
2023-07-26 1:51 ` Dmitry Gutov
2023-07-26 9:09 ` Ihor Radchenko
2023-07-27 0:41 ` Dmitry Gutov
2023-07-27 5:22 ` Eli Zaretskii
2023-07-27 8:20 ` Ihor Radchenko
2023-07-27 8:47 ` Eli Zaretskii
2023-07-27 9:28 ` Ihor Radchenko
2023-07-27 13:30 ` Dmitry Gutov
2023-07-29 0:12 ` Dmitry Gutov [this message]
2023-07-29 6:15 ` Eli Zaretskii
2023-07-30 1:35 ` Dmitry Gutov
2023-07-31 11:38 ` Eli Zaretskii
2023-09-08 0:53 ` Dmitry Gutov
2023-09-08 6:35 ` Eli Zaretskii
2023-09-10 1:30 ` Dmitry Gutov
2023-09-10 5:33 ` Eli Zaretskii
2023-09-11 0:02 ` Dmitry Gutov
2023-09-11 11:57 ` Eli Zaretskii
2023-09-11 23:06 ` Dmitry Gutov
2023-09-12 11:39 ` Eli Zaretskii
2023-09-12 13:11 ` Dmitry Gutov
2023-09-12 14:23 ` Dmitry Gutov
2023-09-12 14:26 ` Dmitry Gutov
2023-09-12 16:32 ` Eli Zaretskii
2023-09-12 18:48 ` Dmitry Gutov
2023-09-12 19:35 ` Eli Zaretskii
2023-09-12 20:27 ` Dmitry Gutov
2023-09-13 11:38 ` Eli Zaretskii
2023-09-13 14:27 ` Dmitry Gutov
2023-09-13 15:07 ` Eli Zaretskii
2023-09-13 17:27 ` Dmitry Gutov
2023-09-13 19:32 ` Eli Zaretskii
2023-09-13 20:38 ` Dmitry Gutov
2023-09-14 5:41 ` Eli Zaretskii
2023-09-16 1:32 ` Dmitry Gutov
2023-09-16 5:37 ` Eli Zaretskii
2023-09-19 19:59 ` bug#66020: (bug#64735 spin-off): regarding the default for read-process-output-max Dmitry Gutov
2023-09-20 11:20 ` Eli Zaretskii
2023-09-21 0:57 ` Dmitry Gutov
2023-09-21 2:36 ` Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors
[not found] ` <58e9135f-915d-beb9-518a-e814ec2a0c5b@gutov.dev>
2023-09-21 13:16 ` Eli Zaretskii
2023-09-21 17:54 ` Dmitry Gutov
2023-09-21 7:42 ` Eli Zaretskii
2023-09-21 14:37 ` Dmitry Gutov
2023-09-21 14:59 ` Eli Zaretskii
2023-09-21 17:40 ` Dmitry Gutov
2023-09-21 18:39 ` Eli Zaretskii
2023-09-21 18:42 ` Dmitry Gutov
2023-09-21 18:49 ` Eli Zaretskii
2023-09-21 17:33 ` Dmitry Gutov
2023-09-23 21:51 ` Dmitry Gutov
2023-09-24 5:29 ` Eli Zaretskii
2024-05-26 15:20 ` Dmitry Gutov
2024-05-26 16:01 ` Eli Zaretskii
2024-05-26 23:27 ` Stefan Kangas
2023-09-21 8:07 ` Stefan Kangas
[not found] ` <b4f2135b-be9d-2423-02ac-9690de8b5a92@gutov.dev>
2023-09-21 13:17 ` Eli Zaretskii
2023-07-25 18:42 ` bug#64735: 29.0.92; find invocations are ~15x slower because of ignores Eli Zaretskii
2023-07-26 1:56 ` Dmitry Gutov
2023-07-26 2:28 ` Eli Zaretskii
2023-07-26 2:35 ` Dmitry Gutov
2023-07-25 19:16 ` sbaugh
2023-07-26 2:28 ` Dmitry Gutov
2023-07-21 2:42 ` Richard Stallman
2023-07-22 2:39 ` Richard Stallman
2023-07-22 5:49 ` Eli Zaretskii
2023-07-22 10:18 ` Ihor Radchenko
2023-07-22 10:42 ` sbaugh
2023-07-22 12:00 ` Eli Zaretskii
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://www.gnu.org/software/emacs/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=59c30342-a7e0-d83b-a128-0faae4cbd633@gutov.dev \
--to=dmitry@gutov.dev \
--cc=64735@debbugs.gnu.org \
--cc=eliz@gnu.org \
--cc=luangruo@yahoo.com \
--cc=sbaugh@janestreet.com \
--cc=yantar92@posteo.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/emacs.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).