From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Dmitry Gutov Newsgroups: gmane.emacs.bugs Subject: bug#66020: (bug#64735 spin-off): regarding the default for read-process-output-max Date: Tue, 19 Sep 2023 22:59:43 +0300 Message-ID: <937d9927-506f-aa36-94e9-3cceb8f629dd@gutov.dev> References: <35f8b664-0241-9f96-1aa0-20ca51b2d34c@gutov.dev> <59c30342-a7e0-d83b-a128-0faae4cbd633@gutov.dev> <83pm4bi6qa.fsf@gnu.org> <83bkfs2tw5.fsf@gnu.org> <18a0b4d8-32bd-3ecd-8db4-32608a1ebba7@gutov.dev> <83il8lxjcu.fsf@gnu.org> <2e21ec81-8e4f-4c02-ea15-43bd6da3daa7@gutov.dev> <8334zmtwwi.fsf@gnu.org> <83tts0rkh5.fsf@gnu.org> <831qf3pd1y.fsf@gnu.org> <28a7916e-92d5-77ab-a61e-f85b59ac76b1@gutov.dev> <83sf7jnq0m.fsf@gnu.org> <5c493f86-0af5-256f-41a7-7d886ab4c5e4@gutov.dev> <83ledanvzw.fsf@gnu.org> <83r0n2m7qz.fsf@gnu.org> <26afa109-9ba3-78a3-0e68-7585ae8e3a19@gutov.dev> <83il8dna30.fsf@gnu.org> <83bke5mhvs.fsf@gnu.org> <83a5tmk79p.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="26735"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.13.0 Cc: 66020@debbugs.gnu.org To: Eli Zaretskii Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Tue Sep 19 22:01:14 2023 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1qiguF-0006jE-Me for geb-bug-gnu-emacs@m.gmane-mx.org; Tue, 19 Sep 2023 22:01:12 +0200 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qigty-0004Ge-N3; Tue, 19 Sep 2023 16:00:54 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qigtx-0004GV-4g for bug-gnu-emacs@gnu.org; Tue, 19 Sep 2023 16:00:53 -0400 Original-Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1qigtw-0000Mf-TH for bug-gnu-emacs@gnu.org; Tue, 19 Sep 2023 16:00:52 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1qigu5-0000yI-MF for bug-gnu-emacs@gnu.org; Tue, 19 Sep 2023 16:01:01 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Dmitry Gutov Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Tue, 19 Sep 2023 20:01:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 66020 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: patch Original-Received: via spool by 66020-submit@debbugs.gnu.org id=B66020.16951536063574 (code B ref 66020); Tue, 19 Sep 2023 20:01:01 +0000 Original-Received: (at 66020) by debbugs.gnu.org; 19 Sep 2023 20:00:06 +0000 Original-Received: from localhost ([127.0.0.1]:57855 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qigtB-0000vY-Fk for submit@debbugs.gnu.org; Tue, 19 Sep 2023 16:00:06 -0400 Original-Received: from out2-smtp.messagingengine.com ([66.111.4.26]:58893) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qigt8-0000uT-Ns for 66020@debbugs.gnu.org; Tue, 19 Sep 2023 16:00:04 -0400 Original-Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.nyi.internal (Postfix) with ESMTP id 3DD6D5C01AA; Tue, 19 Sep 2023 15:59:48 -0400 (EDT) Original-Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Tue, 19 Sep 2023 15:59:48 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc :cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to; s=fm3; t= 1695153588; x=1695239988; bh=O2MS+YZGx/qBn4Xj9GVu4SjO0x3m8JC4VSs qXXmB7Ck=; b=hhuHbUXTvbhwUloJ4wyqHiBXkPI9oLZ4vL35zZ4h6s/MaZ6fXQO YqJt+D69oAYdfWbJV5o+Jd3o48JpixdF3sUa/8R7Th7jsy0z9IYbFxscGnP0RW3H 1CDUQizlw02z1Etfm9mbS1YlVGgUIcuTc3uxVrj+XTwQifUs0y9daacIsKuI/abx ga0er+M3Re8BB0QNDZIyMcv1rtgm3HOSHQ5nY4bmuFI/iTxFzqOFGrb9Ko4gbvW7 m+P7PUcRQtA8Vtu9PMVZYdjH7xzGU9vBljHKOvza3mRPBlV7f+EDHz+8760x+VUW fyiapAm+yDdXCr8f0vQjUS5wMnqvptAek4A== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; t= 1695153588; x=1695239988; bh=O2MS+YZGx/qBn4Xj9GVu4SjO0x3m8JC4VSs qXXmB7Ck=; b=N7A+A7p5uchSAYMz6ojNh61/irnuk2xF62Bj0T96C0dyFQ1zGlJ uA3J24/zDBPxRPBg/mZFeUM891+WGWoub7XsEB9vG50ucj+9aYot/96ivaK5PiOK fHgKNaMeQ/MGX9jQllrOLLpqbYlRrOM1NFF4v1/ei1eUeck45sKR8P7BHjFCGw0c Lm5bv4MrLZoumZq+dULA3dIcCY7ie8jxwhyQfpE3dO/rlHggfQnzW6kz6k/gI+uE TMsQck/JuyyLz0nyTHVgSuraC60wu3Khk5VrFzMrvLMhiATPBzlZyA2cQ9KCoYVG o00uH+5/F6yhUzhsxlQUAuu0Ssh/Fnar5zw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrudekuddguddufecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpefkffggfgfuvfevfhfhjggtgfesthejredttdefjeenucfhrhhomhepffhm ihhtrhihucfiuhhtohhvuceoughmihhtrhihsehguhhtohhvrdguvghvqeenucggtffrrg htthgvrhhnpeeigfetveehveevffehledtueekieeikeeufeegudfgfeeghfdulefgfeev ledvveenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpe gumhhithhrhiesghhuthhovhdruggvvh X-ME-Proxy: Feedback-ID: i0e71465a:Fastmail Original-Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 19 Sep 2023 15:59:47 -0400 (EDT) Content-Language: en-US In-Reply-To: <83a5tmk79p.fsf@gnu.org> X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.bugs:270894 Archived-At: This is another continuation from bug#64735, a subthread in this bug seems more fitting, given that I did most of the tests with its patch applied. On 16/09/2023 08:37, Eli Zaretskii wrote: >> Date: Sat, 16 Sep 2023 04:32:26 +0300 >> Cc:luangruo@yahoo.com,sbaugh@janestreet.com,yantar92@posteo.net, >> 64735@debbugs.gnu.org >> From: Dmitry Gutov >> >>>> I wonder what scenario that might become apparent in. Launching many >>>> small processes at once? Can't think of a realistic test case. >>> One process suffices. The effect might not be significant, but >>> slowdowns due to new features are generally considered regressions. >> We'd need some objective way to evaluate this. Otherwise we'd just stop >> at the prospect of slowing down some process somewhere by 9ns (never >> mind speeding others up). > That could indeed happen, and did happen in other cases. My personal > conclusion from similar situations is that it is impossible to tell in > advance what the reaction will be; we need to present the numbers and > see how the chips fall. I wrote this test: (defun test-ls-output () (with-temp-buffer (let ((proc (make-process :name "ls" :sentinel (lambda (&rest _)) :buffer (current-buffer) :stderr (current-buffer) :connection-type 'pipe :command '("ls")))) (while (accept-process-output proc)) (buffer-string)))) And tried to find some case where the difference is the least in favor of high buffer length. The one in favor of it we already know (a process with lots and lots of output). But when running 'ls' on a small directory (output 500 chars long), the variance in benchmarking is larger than any difference I can see from changing read-process-output-max from 4096 to 40960 (or to 40900 even). The benchmark is the following: (benchmark 1000 '(let ((read-process-output-fast t) (read-process-output-max 4096)) (test-ls-output))) When the directory is a little large (output ~50000 chars), there is more nuance. At first, as long as (!) read_and_insert_process_output_v2 patch is applied and read-process-output-fast is non-nil, the difference is negligible: | read-process-output-max | bench result | | 4096 | (4.566418994 28 0.8000380139999992) | | 40960 | (4.640526664 32 0.8330555910000008) | | 409600 | (4.629948652 30 0.7989731299999994) | For completeness, here are the same results for read-process-output-fast=nil (emacs-29 is similar, though all a little slower): | read-process-output-max | bench result | | 4096 | (4.953397326 52 1.354643750000001) | | 40960 | (6.942334958 75 2.0616055079999995) | | 409600 | (7.124765651 76 2.0892871070000005) | But as the session gets older (and I repeat these and other memory-intensive benchmarks), the outlay changes, and the larger buffer leads to uniformly worse number (the below is taken with read-process-output-fast=t; with that var set to nil the results were even worse): | read-process-output-max | bench result | | 4096 | (5.02324481 41 0.8851443580000051) | | 40960 | (5.438721274 61 1.2202541989999958) | | 409600 | (6.11188183 77 1.5461468160000038) | ...which seems odd given that in general, the buffer length closer to the length of the output should be preferable, because otherwise it is allocated multiple times, and read_process_output is likewise called more. Perhaps longer strings get more difficult to allocate as fragmentation increases? So, the last table is from a session I had running from yesterday, and the first table was produced after I restarted Emacs about an hour ago (the numbers were stable for 1-2 hours while I was writing this email on-and-off, then started degrading again a little bit, though not yet -- a couple of hours since -- even halfway to the numbers in the last table). Where to go from here? - Maybe we declare the difference insignificant and bump the value of read-process-output-max, given that it helps in other cases, - Or try to find out the cause for degradation, - Or keep the default the same, but make it easier to use different value for different processes (meaning, we resurrect the discussion in bug#38561).