From: Eli Zaretskii <eliz@gnu.org>
To: Dmitry Gutov <dmitry@gutov.dev>
Cc: luangruo@yahoo.com, sbaugh@janestreet.com, yantar92@posteo.net,
64735@debbugs.gnu.org
Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores
Date: Fri, 08 Sep 2023 09:35:45 +0300 [thread overview]
Message-ID: <83il8lxjcu.fsf@gnu.org> (raw)
In-Reply-To: <18a0b4d8-32bd-3ecd-8db4-32608a1ebba7@gutov.dev> (message from Dmitry Gutov on Fri, 8 Sep 2023 03:53:37 +0300)
> Date: Fri, 8 Sep 2023 03:53:37 +0300
> Cc: luangruo@yahoo.com, sbaugh@janestreet.com, yantar92@posteo.net,
> 64735@debbugs.gnu.org
> From: Dmitry Gutov <dmitry@gutov.dev>
>
> >> (("with-find 4096" . "Elapsed time: 1.737742s (1.019624s in 28 GCs)")
> >> ("with-find 40960" . "Elapsed time: 1.515376s (0.942906s in 26 GCs)")
> >> ("with-find 409600" . "Elapsed time: 1.458987s (0.948857s in 26 GCs)")
> >> ("with-find 1048576" . "Elapsed time: 1.363882s (0.888599s in 24 GCs)")
> >> ("with-find-p 4096" . "Elapsed time: 1.202522s (0.745758s in 19 GCs)")
> >> ("with-find-p 40960" . "Elapsed time: 1.005221s (0.640815s in 16 GCs)")
> >> ("with-find-p 409600" . "Elapsed time: 0.855483s (0.591208s in 15 GCs)")
> >> ("with-find-p 1048576". "Elapsed time: 0.825936s (0.623876s in 16 GCs)")
> >> ("with-find-sync 4096" . "Elapsed time: 0.848059s (0.272570s in 7 GCs)")
> >> ("with-find-sync 409600"."Elapsed time: 0.912932s (0.339230s in 9 GCs)")
> >> ("with-find-sync 1048576"."Elapsed time: 0.880479s (0.303047s in 8 GCs)"
> >> ))
> >>
> >> What was puzzling for me, overall, is that if we take "with-find 409600"
> >> (the fastest among the asynchronous runs without parallelism) and
> >> "with-find-sync", the difference in GC time (which is repeatable),
> >> 0.66s, almost covers all the difference in performance. And as for
> >> "with-find-p 409600", it would come out on top! Which it did in Ihor's
> >> tests when GC was disabled.
> >>
> >> But where does the extra GC time come from? Is it from extra consing in
> >> the asynchronous call's case? If it is, it's not from all the chunked
> >> strings, apparently, given that increasing max string's size (and
> >> decreasing their number by 2x-6x, according to my logging) doesn't
> >> affect the reported GC time much.
> >>
> >> Could the extra time spent in GC just come from the fact that it's given
> >> more opportunities to run, maybe? call_process stays entirely in C,
> >> whereas make-process, with its asynchronous approach, goes between C and
> >> Lisp even time it receives input. The report above might indicate so:
> >> with-find-p have ~20 garbage collection cycles, whereas with-find-sync -
> >> only ~10. Or could there be some other source of consing, unrelated to
> >> the process output string, and how finely they are sliced?
> >
> > These questions can only be answered by dumping the values of the 2 GC
> > thresholds and of consing_until_gc for each GC cycle. It could be
> > that we are consing more Lisp memory, or it could be that one of the
> > implementations provides fewer opportunities for Emacs to call
> > maybe_gc. Or it could be some combination of the two.
>
> Do you think the outputs of
> https://elpa.gnu.org/packages/emacs-gc-stats.html could help?
I think you'd need to expose consing_until_gc to Lisp, and then you
can collect the data from Lisp.
> Otherwise, I suppose I need to add some fprintf's somewhere. Would the
> beginning of maybe_gc inside lisp.h be a good place for that?
I can only recommend the fprintf method if doing this from Lisp is
impossible for some reason.
> >> If we get back to increasing read-process-output-max, which does help
> >> (apparently due to reducing the number we switch between reading from
> >> the process and doing... whatever else), the sweet spot seems to be
> >> 1048576, which is my system's maximum value. Anything higher - and the
> >> perf goes back to worse -- I'm guessing something somewhere resets the
> >> value to default? Not sure why it doesn't clip to the maximum allowed,
> >> though.
> >>
> >> Anyway, it would be helpful to be able to decide on as high as possible
> >> value without manually reading from /proc/sys/fs/pipe-max-size. And what
> >> of other OSes?
> >
> > Is this with pipes or with PTYs?
>
> All examples which use make-process call it with :connection-type 'pipe.
>
> The one that calls process-file (the "synchronous" impl) also probably
> does, but I don't see that in the docstring.
Yes, call-process uses pipes. So finding the optimum boils down to
running various scenarios. It is also possible that the optimum will
be different on different systems, btw.
next prev parent reply other threads:[~2023-09-08 6:35 UTC|newest]
Thread overview: 202+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-07-19 21:16 bug#64735: 29.0.92; find invocations are ~15x slower because of ignores Spencer Baugh
2023-07-20 5:00 ` Eli Zaretskii
2023-07-20 12:22 ` sbaugh
2023-07-20 12:42 ` Dmitry Gutov
2023-07-20 13:43 ` Spencer Baugh
2023-07-20 18:54 ` Dmitry Gutov
2023-07-20 12:38 ` Dmitry Gutov
2023-07-20 13:20 ` Ihor Radchenko
2023-07-20 15:19 ` Dmitry Gutov
2023-07-20 15:42 ` Ihor Radchenko
2023-07-20 15:57 ` Dmitry Gutov
2023-07-20 16:03 ` Ihor Radchenko
2023-07-20 18:56 ` Dmitry Gutov
2023-07-21 9:14 ` Ihor Radchenko
2023-07-20 16:33 ` Eli Zaretskii
2023-07-20 16:36 ` Ihor Radchenko
2023-07-20 16:45 ` Eli Zaretskii
2023-07-20 17:23 ` Ihor Radchenko
2023-07-20 18:24 ` Eli Zaretskii
2023-07-20 18:29 ` Ihor Radchenko
2023-07-20 18:43 ` Eli Zaretskii
2023-07-20 18:57 ` Ihor Radchenko
2023-07-21 12:37 ` Dmitry Gutov
2023-07-21 12:58 ` Ihor Radchenko
2023-07-21 13:00 ` Dmitry Gutov
2023-07-21 13:34 ` Ihor Radchenko
2023-07-21 13:36 ` Dmitry Gutov
2023-07-21 13:46 ` Ihor Radchenko
2023-07-21 15:41 ` Dmitry Gutov
2023-07-21 15:48 ` Ihor Radchenko
2023-07-21 19:53 ` Dmitry Gutov
2023-07-23 5:40 ` Ihor Radchenko
2023-07-23 11:50 ` Michael Albinus
2023-07-24 7:35 ` Ihor Radchenko
2023-07-24 7:59 ` Michael Albinus
2023-07-24 8:22 ` Ihor Radchenko
2023-07-24 9:31 ` Michael Albinus
2023-07-21 7:45 ` Michael Albinus
2023-07-21 10:46 ` Eli Zaretskii
2023-07-21 11:32 ` Michael Albinus
2023-07-21 11:51 ` Ihor Radchenko
2023-07-21 12:01 ` Michael Albinus
2023-07-21 12:20 ` Ihor Radchenko
2023-07-21 12:25 ` Ihor Radchenko
2023-07-21 12:46 ` Eli Zaretskii
2023-07-21 13:01 ` Michael Albinus
2023-07-21 13:23 ` Ihor Radchenko
2023-07-21 15:31 ` Michael Albinus
2023-07-21 15:38 ` Ihor Radchenko
2023-07-21 15:49 ` Michael Albinus
2023-07-21 15:55 ` Eli Zaretskii
2023-07-21 16:08 ` Michael Albinus
2023-07-21 16:15 ` Ihor Radchenko
2023-07-21 16:38 ` Eli Zaretskii
2023-07-21 16:43 ` Ihor Radchenko
2023-07-21 16:43 ` Michael Albinus
2023-07-21 17:45 ` Eli Zaretskii
2023-07-21 17:55 ` Michael Albinus
2023-07-21 18:38 ` Eli Zaretskii
2023-07-21 19:33 ` Spencer Baugh
2023-07-22 5:27 ` Eli Zaretskii
2023-07-22 10:38 ` sbaugh
2023-07-22 11:58 ` Eli Zaretskii
2023-07-22 14:14 ` Ihor Radchenko
2023-07-22 14:32 ` Eli Zaretskii
2023-07-22 15:07 ` Ihor Radchenko
2023-07-22 15:29 ` Eli Zaretskii
2023-07-23 7:52 ` Ihor Radchenko
2023-07-23 8:01 ` Eli Zaretskii
2023-07-23 8:11 ` Ihor Radchenko
2023-07-23 9:11 ` Eli Zaretskii
2023-07-23 9:34 ` Ihor Radchenko
2023-07-23 9:39 ` Eli Zaretskii
2023-07-23 9:42 ` Ihor Radchenko
2023-07-23 10:20 ` Eli Zaretskii
2023-07-23 11:43 ` Ihor Radchenko
2023-07-23 12:49 ` Eli Zaretskii
2023-07-23 12:57 ` Ihor Radchenko
2023-07-23 13:32 ` Eli Zaretskii
2023-07-23 13:56 ` Ihor Radchenko
2023-07-23 14:32 ` Eli Zaretskii
2023-07-22 17:18 ` sbaugh
2023-07-22 17:26 ` Ihor Radchenko
2023-07-22 17:46 ` Eli Zaretskii
2023-07-22 18:31 ` Eli Zaretskii
2023-07-22 19:06 ` Eli Zaretskii
2023-07-22 20:53 ` Spencer Baugh
2023-07-23 6:15 ` Eli Zaretskii
2023-07-23 7:48 ` Ihor Radchenko
2023-07-23 8:06 ` Eli Zaretskii
2023-07-23 8:16 ` Ihor Radchenko
2023-07-23 9:13 ` Eli Zaretskii
2023-07-23 9:16 ` Ihor Radchenko
2023-07-23 11:44 ` Michael Albinus
2023-07-23 2:59 ` Richard Stallman
2023-07-23 5:28 ` Eli Zaretskii
2023-07-22 8:17 ` Michael Albinus
2023-07-21 13:17 ` Ihor Radchenko
2023-07-21 12:27 ` Michael Albinus
2023-07-21 12:30 ` Ihor Radchenko
2023-07-21 13:04 ` Michael Albinus
2023-07-21 13:24 ` Ihor Radchenko
2023-07-21 15:36 ` Michael Albinus
2023-07-21 15:44 ` Ihor Radchenko
2023-07-21 12:39 ` Eli Zaretskii
2023-07-21 13:09 ` Michael Albinus
2023-07-21 12:38 ` Dmitry Gutov
2023-07-20 17:08 ` Spencer Baugh
2023-07-20 17:24 ` Eli Zaretskii
2023-07-22 6:35 ` Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors
2023-07-20 17:25 ` Ihor Radchenko
2023-07-21 19:31 ` Spencer Baugh
2023-07-21 19:37 ` Ihor Radchenko
2023-07-21 19:56 ` Dmitry Gutov
2023-07-21 20:11 ` Spencer Baugh
2023-07-22 6:39 ` Po Lu via Bug reports for GNU Emacs, the Swiss army knife of text editors
2023-07-22 21:01 ` Dmitry Gutov
2023-07-23 5:11 ` Eli Zaretskii
2023-07-23 10:46 ` Dmitry Gutov
2023-07-23 11:18 ` Eli Zaretskii
2023-07-23 17:46 ` Dmitry Gutov
2023-07-23 17:56 ` Eli Zaretskii
2023-07-23 17:58 ` Dmitry Gutov
2023-07-23 18:21 ` Eli Zaretskii
2023-07-23 19:07 ` Dmitry Gutov
2023-07-23 19:27 ` Eli Zaretskii
2023-07-23 19:44 ` Dmitry Gutov
2023-07-23 19:27 ` Dmitry Gutov
2023-07-24 11:20 ` Eli Zaretskii
2023-07-24 12:55 ` Dmitry Gutov
2023-07-24 13:26 ` Eli Zaretskii
2023-07-25 2:41 ` Dmitry Gutov
2023-07-25 8:22 ` Ihor Radchenko
2023-07-26 1:51 ` Dmitry Gutov
2023-07-26 9:09 ` Ihor Radchenko
2023-07-27 0:41 ` Dmitry Gutov
2023-07-27 5:22 ` Eli Zaretskii
2023-07-27 8:20 ` Ihor Radchenko
2023-07-27 8:47 ` Eli Zaretskii
2023-07-27 9:28 ` Ihor Radchenko
2023-07-27 13:30 ` Dmitry Gutov
2023-07-29 0:12 ` Dmitry Gutov
2023-07-29 6:15 ` Eli Zaretskii
2023-07-30 1:35 ` Dmitry Gutov
2023-07-31 11:38 ` Eli Zaretskii
2023-09-08 0:53 ` Dmitry Gutov
2023-09-08 6:35 ` Eli Zaretskii [this message]
2023-09-10 1:30 ` Dmitry Gutov
2023-09-10 5:33 ` Eli Zaretskii
2023-09-11 0:02 ` Dmitry Gutov
2023-09-11 11:57 ` Eli Zaretskii
2023-09-11 23:06 ` Dmitry Gutov
2023-09-12 11:39 ` Eli Zaretskii
2023-09-12 13:11 ` Dmitry Gutov
2023-09-12 14:23 ` Dmitry Gutov
2023-09-12 14:26 ` Dmitry Gutov
2023-09-12 16:32 ` Eli Zaretskii
2023-09-12 18:48 ` Dmitry Gutov
2023-09-12 19:35 ` Eli Zaretskii
2023-09-12 20:27 ` Dmitry Gutov
2023-09-13 11:38 ` Eli Zaretskii
2023-09-13 14:27 ` Dmitry Gutov
2023-09-13 15:07 ` Eli Zaretskii
2023-09-13 17:27 ` Dmitry Gutov
2023-09-13 19:32 ` Eli Zaretskii
2023-09-13 20:38 ` Dmitry Gutov
2023-09-14 5:41 ` Eli Zaretskii
2023-09-16 1:32 ` Dmitry Gutov
2023-09-16 5:37 ` Eli Zaretskii
2023-09-19 19:59 ` bug#66020: (bug#64735 spin-off): regarding the default for read-process-output-max Dmitry Gutov
2023-09-20 11:20 ` Eli Zaretskii
2023-09-21 0:57 ` Dmitry Gutov
2023-09-21 2:36 ` Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors
[not found] ` <58e9135f-915d-beb9-518a-e814ec2a0c5b@gutov.dev>
2023-09-21 13:16 ` Eli Zaretskii
2023-09-21 17:54 ` Dmitry Gutov
2023-09-21 7:42 ` Eli Zaretskii
2023-09-21 14:37 ` Dmitry Gutov
2023-09-21 14:59 ` Eli Zaretskii
2023-09-21 17:40 ` Dmitry Gutov
2023-09-21 18:39 ` Eli Zaretskii
2023-09-21 18:42 ` Dmitry Gutov
2023-09-21 18:49 ` Eli Zaretskii
2023-09-21 17:33 ` Dmitry Gutov
2023-09-23 21:51 ` Dmitry Gutov
2023-09-24 5:29 ` Eli Zaretskii
2024-05-26 15:20 ` Dmitry Gutov
2024-05-26 16:01 ` Eli Zaretskii
2024-05-26 23:27 ` Stefan Kangas
2023-09-21 8:07 ` Stefan Kangas
[not found] ` <b4f2135b-be9d-2423-02ac-9690de8b5a92@gutov.dev>
2023-09-21 13:17 ` Eli Zaretskii
2023-07-25 18:42 ` bug#64735: 29.0.92; find invocations are ~15x slower because of ignores Eli Zaretskii
2023-07-26 1:56 ` Dmitry Gutov
2023-07-26 2:28 ` Eli Zaretskii
2023-07-26 2:35 ` Dmitry Gutov
2023-07-25 19:16 ` sbaugh
2023-07-26 2:28 ` Dmitry Gutov
2023-07-21 2:42 ` Richard Stallman
2023-07-22 2:39 ` Richard Stallman
2023-07-22 5:49 ` Eli Zaretskii
2023-07-22 10:18 ` Ihor Radchenko
2023-07-22 10:42 ` sbaugh
2023-07-22 12:00 ` Eli Zaretskii
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://www.gnu.org/software/emacs/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=83il8lxjcu.fsf@gnu.org \
--to=eliz@gnu.org \
--cc=64735@debbugs.gnu.org \
--cc=dmitry@gutov.dev \
--cc=luangruo@yahoo.com \
--cc=sbaugh@janestreet.com \
--cc=yantar92@posteo.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/emacs.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).