From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Eli Zaretskii Newsgroups: gmane.emacs.bugs Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores Date: Fri, 08 Sep 2023 09:35:45 +0300 Message-ID: <83il8lxjcu.fsf@gnu.org> References: <5c4d9bea-3eb9-b262-138a-4ea0cb203436@gutov.dev> <87tttypp2e.fsf@localhost> <87r0p030w0.fsf@yahoo.com> <83sf9f6wm0.fsf@gnu.org> <83sf9eub9d.fsf@gnu.org> <2d844a34-857d-3d59-b897-73372baac480@gutov.dev> <83bkg2tsu6.fsf@gnu.org> <83bd4246-ac41-90ec-1df3-02d0bd59ca44@gutov.dev> <834jlttv1p.fsf@gnu.org> <937c3b8e-7742-91b7-c2cf-4cadd0782f0c@gutov.dev> <83a5vlsanw.fsf@gnu.org> <69a98e2a-5816-d36b-9d04-8609291333cd@gutov.dev> <87351cs8no.fsf@localhost> <35163e56-607d-9c5b-e3e8-5d5b548b3cb7@gutov.dev> <878rb3m43b.fsf@localhost> <83v8e6lyi4.fsf@gnu.org> <35f8b664-0241-9f96-1aa0-20ca51b2d34c@gutov.dev> <59c30342-a7e0-d83b-a128-0faae4cbd633@gutov.dev> <83pm4bi6qa.fsf@gnu.org> <83bkfs2tw5.fsf@gnu.org> <18a0b4d8-32bd-3ecd-8db4-32608a1ebba7@gutov.dev> Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="12490"; mail-complaints-to="usenet@ciao.gmane.io" Cc: luangruo@yahoo.com, sbaugh@janestreet.com, yantar92@posteo.net, 64735@debbugs.gnu.org To: Dmitry Gutov Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Fri Sep 08 08:37:25 2023 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1qeV7N-00034C-IR for geb-bug-gnu-emacs@m.gmane-mx.org; Fri, 08 Sep 2023 08:37:25 +0200 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qeV73-0007lq-75; Fri, 08 Sep 2023 02:37:05 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qeV6x-0007lW-V4 for bug-gnu-emacs@gnu.org; Fri, 08 Sep 2023 02:37:00 -0400 Original-Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1qeV6x-0007n3-Mc for bug-gnu-emacs@gnu.org; Fri, 08 Sep 2023 02:36:59 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1qeV6z-00036l-OQ for bug-gnu-emacs@gnu.org; Fri, 08 Sep 2023 02:37:01 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Eli Zaretskii Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Fri, 08 Sep 2023 06:37:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 64735 X-GNU-PR-Package: emacs Original-Received: via spool by 64735-submit@debbugs.gnu.org id=B64735.169415498411898 (code B ref 64735); Fri, 08 Sep 2023 06:37:01 +0000 Original-Received: (at 64735) by debbugs.gnu.org; 8 Sep 2023 06:36:24 +0000 Original-Received: from localhost ([127.0.0.1]:42030 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qeV6N-00035q-HJ for submit@debbugs.gnu.org; Fri, 08 Sep 2023 02:36:24 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:37808) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qeV6J-00035X-AY for 64735@debbugs.gnu.org; Fri, 08 Sep 2023 02:36:21 -0400 Original-Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qeV6B-0007SI-7z; Fri, 08 Sep 2023 02:36:11 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date: mime-version; bh=rlsXkYvfXeSdQet6A3eWOCfH2P3C49eUVv3ZL7mxz/0=; b=Szbrau3yHD2L OmDsJZmh7ebyYqOrRUkQN1zunnAZdgkkSD0Vejw15meINIrNwb8LWHP4gk0Kzl1UKqbeWoUMd5Q1J 0Hc7Gqwi3rh0ya0pBUNBj8wxMx3dxVPwZOxFPIZprvDw5NtgekBhFrm/rxAzpLlvlhtPhuwMJ+oJv K3cJkYiXjPFi7Bjfn39awRHBobOhIOvIF/2Xl7ybo9+W1b8IWDbGwAZgW1/Vuzj6WExmPf9SdvoC5 wmo7oZcTTXK7PBBIgW1rmjiQEJ0wRP6McchSFnfUp72U6js7Jo7whu4cBOT74DBMhZy7VHVNWGXsq +hJ4CQP09McT/kSujXSxRw==; In-Reply-To: <18a0b4d8-32bd-3ecd-8db4-32608a1ebba7@gutov.dev> (message from Dmitry Gutov on Fri, 8 Sep 2023 03:53:37 +0300) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.bugs:269727 Archived-At: > Date: Fri, 8 Sep 2023 03:53:37 +0300 > Cc: luangruo@yahoo.com, sbaugh@janestreet.com, yantar92@posteo.net, > 64735@debbugs.gnu.org > From: Dmitry Gutov > > >> (("with-find 4096" . "Elapsed time: 1.737742s (1.019624s in 28 GCs)") > >> ("with-find 40960" . "Elapsed time: 1.515376s (0.942906s in 26 GCs)") > >> ("with-find 409600" . "Elapsed time: 1.458987s (0.948857s in 26 GCs)") > >> ("with-find 1048576" . "Elapsed time: 1.363882s (0.888599s in 24 GCs)") > >> ("with-find-p 4096" . "Elapsed time: 1.202522s (0.745758s in 19 GCs)") > >> ("with-find-p 40960" . "Elapsed time: 1.005221s (0.640815s in 16 GCs)") > >> ("with-find-p 409600" . "Elapsed time: 0.855483s (0.591208s in 15 GCs)") > >> ("with-find-p 1048576". "Elapsed time: 0.825936s (0.623876s in 16 GCs)") > >> ("with-find-sync 4096" . "Elapsed time: 0.848059s (0.272570s in 7 GCs)") > >> ("with-find-sync 409600"."Elapsed time: 0.912932s (0.339230s in 9 GCs)") > >> ("with-find-sync 1048576"."Elapsed time: 0.880479s (0.303047s in 8 GCs)" > >> )) > >> > >> What was puzzling for me, overall, is that if we take "with-find 409600" > >> (the fastest among the asynchronous runs without parallelism) and > >> "with-find-sync", the difference in GC time (which is repeatable), > >> 0.66s, almost covers all the difference in performance. And as for > >> "with-find-p 409600", it would come out on top! Which it did in Ihor's > >> tests when GC was disabled. > >> > >> But where does the extra GC time come from? Is it from extra consing in > >> the asynchronous call's case? If it is, it's not from all the chunked > >> strings, apparently, given that increasing max string's size (and > >> decreasing their number by 2x-6x, according to my logging) doesn't > >> affect the reported GC time much. > >> > >> Could the extra time spent in GC just come from the fact that it's given > >> more opportunities to run, maybe? call_process stays entirely in C, > >> whereas make-process, with its asynchronous approach, goes between C and > >> Lisp even time it receives input. The report above might indicate so: > >> with-find-p have ~20 garbage collection cycles, whereas with-find-sync - > >> only ~10. Or could there be some other source of consing, unrelated to > >> the process output string, and how finely they are sliced? > > > > These questions can only be answered by dumping the values of the 2 GC > > thresholds and of consing_until_gc for each GC cycle. It could be > > that we are consing more Lisp memory, or it could be that one of the > > implementations provides fewer opportunities for Emacs to call > > maybe_gc. Or it could be some combination of the two. > > Do you think the outputs of > https://elpa.gnu.org/packages/emacs-gc-stats.html could help? I think you'd need to expose consing_until_gc to Lisp, and then you can collect the data from Lisp. > Otherwise, I suppose I need to add some fprintf's somewhere. Would the > beginning of maybe_gc inside lisp.h be a good place for that? I can only recommend the fprintf method if doing this from Lisp is impossible for some reason. > >> If we get back to increasing read-process-output-max, which does help > >> (apparently due to reducing the number we switch between reading from > >> the process and doing... whatever else), the sweet spot seems to be > >> 1048576, which is my system's maximum value. Anything higher - and the > >> perf goes back to worse -- I'm guessing something somewhere resets the > >> value to default? Not sure why it doesn't clip to the maximum allowed, > >> though. > >> > >> Anyway, it would be helpful to be able to decide on as high as possible > >> value without manually reading from /proc/sys/fs/pipe-max-size. And what > >> of other OSes? > > > > Is this with pipes or with PTYs? > > All examples which use make-process call it with :connection-type 'pipe. > > The one that calls process-file (the "synchronous" impl) also probably > does, but I don't see that in the docstring. Yes, call-process uses pipes. So finding the optimum boils down to running various scenarios. It is also possible that the optimum will be different on different systems, btw.