From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Eli Zaretskii Newsgroups: gmane.emacs.bugs Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores Date: Mon, 31 Jul 2023 14:38:18 +0300 Message-ID: <83bkfs2tw5.fsf@gnu.org> References: <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@gutov.dev> <87wmyupvlw.fsf@localhost> <5c4d9bea-3eb9-b262-138a-4ea0cb203436@gutov.dev> <87tttypp2e.fsf@localhost> <87r0p030w0.fsf@yahoo.com> <83sf9f6wm0.fsf@gnu.org> <83sf9eub9d.fsf@gnu.org> <2d844a34-857d-3d59-b897-73372baac480@gutov.dev> <83bkg2tsu6.fsf@gnu.org> <83bd4246-ac41-90ec-1df3-02d0bd59ca44@gutov.dev> <834jlttv1p.fsf@gnu.org> <937c3b8e-7742-91b7-c2cf-4cadd0782f0c@gutov.dev> <83a5vlsanw.fsf@gnu.org> <69a98e2a-5816-d36b-9d04-8609291333cd@gutov.dev> <87351cs8no.fsf@localhost> <35163e56-607d-9c5b-e3e8-5d5b548b3cb7@gutov.dev> <878rb3m43b.fsf@localhost> <83v8e6lyi4.fsf@gnu.org> <35f8b664-0241-9f96-1aa0-20ca51b2d34c@gutov.dev> <59c30342-a7e0-d83b-a128-0faae4cbd633@gutov.dev> <83pm4bi6qa.fsf@gnu.org> Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="363"; mail-complaints-to="usenet@ciao.gmane.io" Cc: luangruo@yahoo.com, sbaugh@janestreet.com, yantar92@posteo.net, 64735@debbugs.gnu.org To: Dmitry Gutov Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Mon Jul 31 13:39:23 2023 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1qQRFC-000AX1-Su for geb-bug-gnu-emacs@m.gmane-mx.org; Mon, 31 Jul 2023 13:39:23 +0200 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qQRF0-0002Qq-Bw; Mon, 31 Jul 2023 07:39:11 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qQREt-0002QY-0f for bug-gnu-emacs@gnu.org; Mon, 31 Jul 2023 07:39:03 -0400 Original-Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1qQREs-00047e-OK for bug-gnu-emacs@gnu.org; Mon, 31 Jul 2023 07:39:02 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1qQREs-0002d7-8Y for bug-gnu-emacs@gnu.org; Mon, 31 Jul 2023 07:39:02 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Eli Zaretskii Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Mon, 31 Jul 2023 11:39:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 64735 X-GNU-PR-Package: emacs Original-Received: via spool by 64735-submit@debbugs.gnu.org id=B64735.169080350810070 (code B ref 64735); Mon, 31 Jul 2023 11:39:02 +0000 Original-Received: (at 64735) by debbugs.gnu.org; 31 Jul 2023 11:38:28 +0000 Original-Received: from localhost ([127.0.0.1]:51935 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qQREJ-0002cL-P6 for submit@debbugs.gnu.org; Mon, 31 Jul 2023 07:38:28 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:36488) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qQREH-0002c9-8I for 64735@debbugs.gnu.org; Mon, 31 Jul 2023 07:38:26 -0400 Original-Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qQREA-00043j-MV; Mon, 31 Jul 2023 07:38:18 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date: mime-version; bh=e3t9o9cqVCQ4loUKEOavC7d3QlIDyiAJPPlZ64FunRM=; b=Zo7XCgOthxeh 1Y3ZfXbGPvOttiM/m1fvCB0mlxT7PzMZiKXvcochspr+2Xf8Os2EYzlFQB5P6qnkHLlDDGbk3uT5L xlmGdMlPAHI+1aF/taopwAFYSJ1xpCj6GiPj7bTEtLatFNSYaJtMuSvBcETsxI61whgE6HGoJdu0X 1DkWfF5JsJ+7/smra/QfsXZShxC0w0vjiuhe1TwUIdY4pxZ8ubr815O/O2uXjF/kSZoFwLkbhAjm+ Gd9RDwYlRjs6zX8d8suECn+ncsjMNaPj6AqWoYSfeIWh48Ii3sbt/F+Y1h6Tt7VI23drkb9yevu5w gC13C55q1IeVGLFSUcIz9g==; Original-Received: from [87.69.77.57] (helo=home-c4e4a596f7) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qQRE9-0006ZH-UE; Mon, 31 Jul 2023 07:38:18 -0400 In-Reply-To: (message from Dmitry Gutov on Sun, 30 Jul 2023 04:35:49 +0300) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.bugs:266407 Archived-At: > Date: Sun, 30 Jul 2023 04:35:49 +0300 > Cc: luangruo@yahoo.com, sbaugh@janestreet.com, yantar92@posteo.net, > 64735@debbugs.gnu.org > From: Dmitry Gutov > > In this context, junks objects are objects that will need to be > collected by garbage collector very soon because they are just a > byproduct of a function's execution (but aren't used in the return > value, for example). The more of them a function creates, the more work > it will be, supposedly, for the GC. > > Heap is perhaps the wrong term (given that C has its own notion of > heap), but I meant the memory managed by the Lisp runtime. > > And chunks are the buffered strings that get passed to the process > filter. Chunks of the process' output. By default, these chunks are 4096 > characters long, but the comparisons tweak that value by 10x and 100x. If the subprocess output is inserted into a buffer, its effect on the GC will be different. (Not sure if this is relevant to the issue at hand, as I lost track of the many variants of the function that were presented.) > > If I read what you wrote superficially, without delving into the > > details (which I can't understand), you are saying that the overall > > amount of consing is roughly the same. > > What is "amount of consing"? Is it just the number of objects? Or does > their size (e.g. string length) affect GC pressure as well? In general, both, since we have 2 GC thresholds, and GC is actually done when both are exceeded. So the effect will also depend on how much Lisp memory is already allocated in the Emacs process where these benchmarks are run. > > This is consistent with the > > fact that the GC times change only very little. So I don't think I > > see, on this level, what puzzles you in this picture. > > Now that you pointed that out, the picture is just more puzzling. While > 0.1s in GC is not insignificant (it's 10% of the whole runtime), it does > seem to have been more of a fluke, and on average the fluctuations in GC > time are smaller. > > Here's an extended comparison: > > (("with-find 4096" . "Elapsed time: 1.737742s (1.019624s in 28 GCs)") > ("with-find 40960" . "Elapsed time: 1.515376s (0.942906s in 26 GCs)") > ("with-find 409600" . "Elapsed time: 1.458987s (0.948857s in 26 GCs)") > ("with-find 1048576" . "Elapsed time: 1.363882s (0.888599s in 24 GCs)") > ("with-find-p 4096" . "Elapsed time: 1.202522s (0.745758s in 19 GCs)") > ("with-find-p 40960" . "Elapsed time: 1.005221s (0.640815s in 16 GCs)") > ("with-find-p 409600" . "Elapsed time: 0.855483s (0.591208s in 15 GCs)") > ("with-find-p 1048576". "Elapsed time: 0.825936s (0.623876s in 16 GCs)") > ("with-find-sync 4096" . "Elapsed time: 0.848059s (0.272570s in 7 GCs)") > ("with-find-sync 409600"."Elapsed time: 0.912932s (0.339230s in 9 GCs)") > ("with-find-sync 1048576"."Elapsed time: 0.880479s (0.303047s in 8 GCs)" > )) > > What was puzzling for me, overall, is that if we take "with-find 409600" > (the fastest among the asynchronous runs without parallelism) and > "with-find-sync", the difference in GC time (which is repeatable), > 0.66s, almost covers all the difference in performance. And as for > "with-find-p 409600", it would come out on top! Which it did in Ihor's > tests when GC was disabled. > > But where does the extra GC time come from? Is it from extra consing in > the asynchronous call's case? If it is, it's not from all the chunked > strings, apparently, given that increasing max string's size (and > decreasing their number by 2x-6x, according to my logging) doesn't > affect the reported GC time much. > > Could the extra time spent in GC just come from the fact that it's given > more opportunities to run, maybe? call_process stays entirely in C, > whereas make-process, with its asynchronous approach, goes between C and > Lisp even time it receives input. The report above might indicate so: > with-find-p have ~20 garbage collection cycles, whereas with-find-sync - > only ~10. Or could there be some other source of consing, unrelated to > the process output string, and how finely they are sliced? These questions can only be answered by dumping the values of the 2 GC thresholds and of consing_until_gc for each GC cycle. It could be that we are consing more Lisp memory, or it could be that one of the implementations provides fewer opportunities for Emacs to call maybe_gc. Or it could be some combination of the two. > If we get back to increasing read-process-output-max, which does help > (apparently due to reducing the number we switch between reading from > the process and doing... whatever else), the sweet spot seems to be > 1048576, which is my system's maximum value. Anything higher - and the > perf goes back to worse -- I'm guessing something somewhere resets the > value to default? Not sure why it doesn't clip to the maximum allowed, > though. > > Anyway, it would be helpful to be able to decide on as high as possible > value without manually reading from /proc/sys/fs/pipe-max-size. And what > of other OSes? Is this with pipes or with PTYs?