From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Dmitry Gutov Newsgroups: gmane.emacs.bugs Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores Date: Tue, 12 Sep 2023 23:27:49 +0300 Message-ID: <5c493f86-0af5-256f-41a7-7d886ab4c5e4@gutov.dev> References: <83bd4246-ac41-90ec-1df3-02d0bd59ca44@gutov.dev> <834jlttv1p.fsf@gnu.org> <937c3b8e-7742-91b7-c2cf-4cadd0782f0c@gutov.dev> <83a5vlsanw.fsf@gnu.org> <69a98e2a-5816-d36b-9d04-8609291333cd@gutov.dev> <87351cs8no.fsf@localhost> <35163e56-607d-9c5b-e3e8-5d5b548b3cb7@gutov.dev> <878rb3m43b.fsf@localhost> <83v8e6lyi4.fsf@gnu.org> <35f8b664-0241-9f96-1aa0-20ca51b2d34c@gutov.dev> <59c30342-a7e0-d83b-a128-0faae4cbd633@gutov.dev> <83pm4bi6qa.fsf@gnu.org> <83bkfs2tw5.fsf@gnu.org> <18a0b4d8-32bd-3ecd-8db4-32608a1ebba7@gutov.dev> <83il8lxjcu.fsf@gnu.org> <2e21ec81-8e4f-4c02-ea15-43bd6da3daa7@gutov.dev> <8334zmtwwi.fsf@gnu.org> <83tts0rkh5.fsf@gnu.org> <831qf3pd1y.fsf@gnu.org> <28a7916e-92d5-77ab-a61e-f85b59ac76b1@gutov.dev> <83sf7jnq0m.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="40640"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.13.0 Cc: luangruo@yahoo.com, sbaugh@janestreet.com, yantar92@posteo.net, 64735@debbugs.gnu.org To: Eli Zaretskii Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Tue Sep 12 22:29:27 2023 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1qgA0j-000AMG-RK for geb-bug-gnu-emacs@m.gmane-mx.org; Tue, 12 Sep 2023 22:29:27 +0200 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qgA0M-00009v-1j; Tue, 12 Sep 2023 16:29:02 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qgA0J-00009T-2m for bug-gnu-emacs@gnu.org; Tue, 12 Sep 2023 16:28:59 -0400 Original-Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1qgA0H-0001hr-Rj for bug-gnu-emacs@gnu.org; Tue, 12 Sep 2023 16:28:58 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1qgA0M-0002Ik-Kj for bug-gnu-emacs@gnu.org; Tue, 12 Sep 2023 16:29:02 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Dmitry Gutov Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Tue, 12 Sep 2023 20:29:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 64735 X-GNU-PR-Package: emacs Original-Received: via spool by 64735-submit@debbugs.gnu.org id=B64735.16945504878781 (code B ref 64735); Tue, 12 Sep 2023 20:29:02 +0000 Original-Received: (at 64735) by debbugs.gnu.org; 12 Sep 2023 20:28:07 +0000 Original-Received: from localhost ([127.0.0.1]:60141 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qg9zS-0002HZ-Pj for submit@debbugs.gnu.org; Tue, 12 Sep 2023 16:28:07 -0400 Original-Received: from out5-smtp.messagingengine.com ([66.111.4.29]:59149) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qg9zP-0002H3-E2 for 64735@debbugs.gnu.org; Tue, 12 Sep 2023 16:28:05 -0400 Original-Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.nyi.internal (Postfix) with ESMTP id A32465C01E8; Tue, 12 Sep 2023 16:27:52 -0400 (EDT) Original-Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Tue, 12 Sep 2023 16:27:52 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc :cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to; s=fm3; t= 1694550472; x=1694636872; bh=GpU9UjNNZ2rCNxsVAsYpUrwlWi5xiIiqvyH gHpFpMzY=; b=de3WkuPF7jTAzFqET5DYh7rVgda2jXwi0vZWMACNX0gYIuKlBKI 1BgJ6aKJNFy8mjre5dvoLb1GkiINlRZxWbbK3QHZ8r72KZiZCgvDHuw6etfaDH/t uepMZqc6Ho/9WnP6t4CBQmuF8DTMuZDe1rm20YkheWSNYYaxiqtPjDNOi9ujvPB/ Afr5+WKrhmHxQeYh0kkawSMi3RZNYAx/uSIgVPTHWMjlQXH+pGr6+VUV1S1L1RR8 hmxFENpWBf2eKg9gsELARiS8PUs4ZEITIUG0IQo1l9tu7y/4mCR9m71cokBEt8xv lFl8cEDuHOETCGssiLxtReCj2e6bk2wE9OQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t= 1694550472; x=1694636872; bh=GpU9UjNNZ2rCNxsVAsYpUrwlWi5xiIiqvyH gHpFpMzY=; b=z3Gyux7C8NG2xZLIHaTEOdgwJYAJNJVp2fCF3oIxtH0CKHFcHRl pyai1QAI/YLyt7qsuTa7TwMZe9XUCUtHDdUu6FWewJzNMXv5h9fcVfR4gub90Km1 L3Vw3468stMtsGrdHL6z/JO9uYY45FHkkj+P40bxRnuw8zYjZ8lpXlfoc09gfMZ8 Z2fAWb3mqI94+amX3Zy6FshlV9hkGneHjXM6Ntmdzz8jWgT4qEO2wdl+ageC8SAB 0yd7SvrblPSiENtUsFJYvUWLW/xI+XbNjrrk8RyEx/FYQdHc6ZMQpEg5+QbzpIZF koqFs8vdQru/sR0f0trhaEKZ5FI7jZEf1Pw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrudeiiedgudehtdcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpefkffggfgfuvfevfhfhjggtgfesthejredttdefjeenucfhrhhomhepffhm ihhtrhihucfiuhhtohhvuceoughmihhtrhihsehguhhtohhvrdguvghvqeenucggtffrrg htthgvrhhnpeeigfetveehveevffehledtueekieeikeeufeegudfgfeeghfdulefgfeev ledvveenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpe gumhhithhrhiesghhuthhovhdruggvvh X-ME-Proxy: Feedback-ID: i0e71465a:Fastmail Original-Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 12 Sep 2023 16:27:51 -0400 (EDT) Content-Language: en-US In-Reply-To: <83sf7jnq0m.fsf@gnu.org> X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.bugs:270219 Archived-At: On 12/09/2023 22:35, Eli Zaretskii wrote: >> Date: Tue, 12 Sep 2023 21:48:37 +0300 >> Cc: luangruo@yahoo.com, sbaugh@janestreet.com, yantar92@posteo.net, >> 64735@debbugs.gnu.org >> From: Dmitry Gutov >> >>> then we could try extending >>> internal-default-process-filter (or writing a new filter function >>> similar to it) so that it inserts the stuff into the gap and then uses >>> decode_coding_gap, >> >> Can that work at all? By the time internal-default-process-filter is >> called, we have already turned the string from char* into Lisp_Object >> text, which we then pass to it. So consing has already happened, IIUC. > > That's why I said "or writing a new filter function". > read_and_dispose_of_process_output will have to call this new filter > differently, passing it the raw text read from the subprocess, where > read_and_dispose_of_process_output current first decodes the text and > produces a Lisp string from it. Then the filter would need to do > something similar to what insert-file-contents does: insert the raw > input into the gap, then call decode_coding_gap to decode that > in-place. Does the patch from my last patch-bearing email look similar enough to what you're describing? The one called read_and_insert_process_output.diff The result there, though, is that a "filter" (in the sense that make-process uses that term) is not used at all. >>> which converts inserted bytes in-place -- that, at >>> least, will be correct and will avoid consing intermediate temporary >>> strings from the process output, then decoding them, then inserting >>> them. Other than that, the -2 and -3 variants are very close >>> runners-up of -5, so maybe I'm missing something, but I see no reason >>> be too excited here? I mean, 0.89 vs 0.92? really? >> >> The important part is not 0.89 vs 0.92 (that would be meaningless >> indeed), but that we have an _asyncronous_ implementation of the feature >> that works as fast as the existing synchronous one (or faster! if we >> also bind read-process-output-max to a large value, the time is 0.72). >> >> The possible applications for that range from simple (printing progress >> bar while the scan is happening) to more advanced (launching a >> concurrent process where we pipe the received file names concurrently to >> 'xargs grep'), including visuals (xref buffer which shows the >> intermediate search results right away, updating them gradually, all >> without blocking the UI). > > Hold your horses. Emacs only reads output from sub-processes when > it's idle. So printing a progress bar (which makes Emacs not idle) > with the asynchronous implementation is basically the same as having > the synchronous implementation call some callback from time to time > (which will then show the progress). Obviously there is more work to be done, including further desgin and benchmarking. But unlike before, at least the starting performance (before further features are added) is not worse. Note that the variant -5 is somewhat limited since it doesn't use a filter - that means that no callbacks a issued while the output is arriving, meaning that if it's taken as base, whatever refreshes would have to be initiated from somewhere else. E.g. from a timer. > As for piping to another process, this is best handled by using a > shell pipe, without passing stuff through Emacs. And even if you do > need to pass it through Emacs, you could do the same with the > synchronous implementation -- only the "xargs" part needs to be > asynchronous, the part that reads file names does not. Right? Yes and no: if both steps are asynchronous, the final output window could be displayed right away, rather than waiting for the first step (or both) to be finished. Which can be a meaningful improvement for some (and still is an upside of 'M-x rgrep'). > Please note: I'm not saying that the asynchronous implementation is > not interesting. It might even have advantages in some specific use > cases. So it is good to have it. It just isn't a breakthrough, > that's all. Not a breakthrough, of course, just a lower-level insight (hopefully). I do think it would be meaningful to manage to reduce the runtime of a real-life program (which includes other work) by 10-20% solely by reducing GC pressure in a generic facility like process output handling. > And if we want to use it in production, we should > probably work on adding that special default filter which inserts and > decodes directly into the buffer, because that will probably lower the > GC pressure and thus has hope of being faster. Or even replace the > default filter implementation with that new one. But a filter must be a Lisp function, which can't help but accept only Lisp strings (not C string) as argument. Isn't that right?