From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Dmitry Gutov Newsgroups: gmane.emacs.bugs Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores Date: Tue, 12 Sep 2023 21:48:37 +0300 Message-ID: <28a7916e-92d5-77ab-a61e-f85b59ac76b1@gutov.dev> References: <2d844a34-857d-3d59-b897-73372baac480@gutov.dev> <83bkg2tsu6.fsf@gnu.org> <83bd4246-ac41-90ec-1df3-02d0bd59ca44@gutov.dev> <834jlttv1p.fsf@gnu.org> <937c3b8e-7742-91b7-c2cf-4cadd0782f0c@gutov.dev> <83a5vlsanw.fsf@gnu.org> <69a98e2a-5816-d36b-9d04-8609291333cd@gutov.dev> <87351cs8no.fsf@localhost> <35163e56-607d-9c5b-e3e8-5d5b548b3cb7@gutov.dev> <878rb3m43b.fsf@localhost> <83v8e6lyi4.fsf@gnu.org> <35f8b664-0241-9f96-1aa0-20ca51b2d34c@gutov.dev> <59c30342-a7e0-d83b-a128-0faae4cbd633@gutov.dev> <83pm4bi6qa.fsf@gnu.org> <83bkfs2tw5.fsf@gnu.org> <18a0b4d8-32bd-3ecd-8db4-32608a1ebba7@gutov.dev> <83il8lxjcu.fsf@gnu.org> <2e21ec81-8e4f-4c02-ea15-43bd6da3daa7@gutov.dev> <8334zmtwwi.fsf@gnu.org> <83tts0rkh5.fsf@gnu.org> <831qf3pd1y.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="8624"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.13.0 Cc: luangruo@yahoo.com, sbaugh@janestreet.com, yantar92@posteo.net, 64735@debbugs.gnu.org To: Eli Zaretskii Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Tue Sep 12 20:49:43 2023 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1qg8SE-0001yX-6s for geb-bug-gnu-emacs@m.gmane-mx.org; Tue, 12 Sep 2023 20:49:42 +0200 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qg8Rx-0002Ax-RZ; Tue, 12 Sep 2023 14:49:30 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qg8RV-00026e-Ps for bug-gnu-emacs@gnu.org; Tue, 12 Sep 2023 14:48:58 -0400 Original-Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1qg8RV-0005Qf-HD for bug-gnu-emacs@gnu.org; Tue, 12 Sep 2023 14:48:57 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1qg8Ra-0002wi-8Z for bug-gnu-emacs@gnu.org; Tue, 12 Sep 2023 14:49:02 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Dmitry Gutov Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Tue, 12 Sep 2023 18:49:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 64735 X-GNU-PR-Package: emacs Original-Received: via spool by 64735-submit@debbugs.gnu.org id=B64735.169454453611311 (code B ref 64735); Tue, 12 Sep 2023 18:49:02 +0000 Original-Received: (at 64735) by debbugs.gnu.org; 12 Sep 2023 18:48:56 +0000 Original-Received: from localhost ([127.0.0.1]:60054 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qg8RT-0002wM-9H for submit@debbugs.gnu.org; Tue, 12 Sep 2023 14:48:55 -0400 Original-Received: from out5-smtp.messagingengine.com ([66.111.4.29]:58265) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qg8RQ-0002vo-7B for 64735@debbugs.gnu.org; Tue, 12 Sep 2023 14:48:54 -0400 Original-Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id 48F855C0186; Tue, 12 Sep 2023 14:48:42 -0400 (EDT) Original-Received: from mailfrontend2 ([10.202.2.163]) by compute4.internal (MEProxy); Tue, 12 Sep 2023 14:48:42 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc :cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to; s=fm3; t= 1694544522; x=1694630922; bh=5y1qtQbgVfNrNKwFJo2jBPpTbQGgQXAYB5Q OS8leH/U=; b=HWKaG07aW/f1tgLXn2Ug6+SwKe4ltw4b4sIbXA25OLFw60gIKhG 76jSeIRaHhuDymhoin2oPcSOaHJEEYyMrhv7unJLD4a6uUjicfRqZV7zxYvhavJM 0e1yc/28xMRJEhHlwIo68yt0cmuwcTa1wRaqA4AeYXZFMy7MMAac+Kzr5shBs7Lu c+43so436oAtjeDziTJHmEDUTp12G0aef591An84mD6V7jJT/ovZ/HA1Eo05Kcef 3eo8YxY59AYhs9b0kh4uugHnqBuRNFqY5oSooCnXEixfU0tRRlRJBimbGv7JoHA3 crEhPCFTNO1nYUH3bIKvQ5tgKIIG+lQ0GNw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:sender:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t= 1694544522; x=1694630922; bh=5y1qtQbgVfNrNKwFJo2jBPpTbQGgQXAYB5Q OS8leH/U=; b=U14jQBM+z728FKFDRsZvSrKwU+N3/J3SmgmFi3Z+1aBX+F/yPGG lq20G/0/LvobW54VJbS2gt32R5lYUPViqQC90UBFZcvtbIzOXkrkl4RxJ2IjWJWO 1/zeb/6vHhC0l+VP3ViAYT/R+Iblhh/A3uPHvq6BRKe326+CxSYISKrFqEKqirt5 0nmDS8tRhXm3Hnb2OOrWsrYxbvINhTm4lkOUOPuOr2PKrQ2WHgGQ9FtWMuG7WgNB IWYG28st632nGt4s9zXxrP3oOKQF82cTWuWwmFT+/Jsm/Lhs3c64rGVQUfUpm4pk mrzUStNNzJNWkUxmyMq8U7Wec0zaJTWOVgg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrudeiiedgudeftdcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmd enucfjughrpefkffggfgfuvfevfhfhjggtgfesthejredttdefjeenucfhrhhomhepffhm ihhtrhihucfiuhhtohhvuceoughmihhtrhihsehguhhtohhvrdguvghvqeenucggtffrrg htthgvrhhnpeeigfetveehveevffehledtueekieeikeeufeegudfgfeeghfdulefgfeev ledvveenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpe gumhhithhrhiesghhuthhovhdruggvvh X-ME-Proxy: Feedback-ID: i0e71465a:Fastmail Original-Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 12 Sep 2023 14:48:40 -0400 (EDT) Content-Language: en-US In-Reply-To: <831qf3pd1y.fsf@gnu.org> X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.bugs:270212 Archived-At: On 12/09/2023 19:32, Eli Zaretskii wrote: >> Date: Tue, 12 Sep 2023 17:23:53 +0300 >> Cc: luangruo@yahoo.com, sbaugh@janestreet.com, yantar92@posteo.net, >> 64735@debbugs.gnu.org >> From: Dmitry Gutov >> >> - Dropping most of the setup in read_and_dispose_of_process_output >> (which creates some consing too) and calling >> Finternal_default_process_filter directly (call_filter_directly.diff), >> when it is the filter to be used anyway. >> >> - Going around that function entirely, skipping the creation of a Lisp >> string (CHARS -> TEXT) and inserting into the buffer directly (when the >> filter is set to the default, of course). Copied and adapted some code >> from 'call_process' for that (read_and_insert_process_output.diff). >> >> Neither are intended as complete proposals, but here are some >> comparisons. Note that either of these patches could only help the >> implementations that don't set up process filter (the naive first one, >> and the new parallel number 5 above). >> >> For testing, I used two different repo checkouts that are large enough >> to not finish too quickly: gecko-dev and torvalds-linux. >> >> master >> >> | Function | gecko-dev | linux | >> | find-directory-files-recursively | 1.69 | 0.41 | >> | find-directory-files-recursively-2 | 1.16 | 0.28 | >> | find-directory-files-recursively-3 | 0.92 | 0.23 | >> | find-directory-files-recursively-5 | 1.07 | 0.26 | >> | find-directory-files-recursively (rpom 409600) | 1.42 | 0.35 | >> | find-directory-files-recursively-2 (rpom 409600) | 0.90 | 0.25 | >> | find-directory-files-recursively-5 (rpom 409600) | 0.89 | 0.24 | >> >> call_filter_directly.diff (basically, not much difference) >> >> | Function | gecko-dev | linux | >> | find-directory-files-recursively | 1.64 | 0.38 | >> | find-directory-files-recursively-5 | 1.05 | 0.26 | >> | find-directory-files-recursively (rpom 409600) | 1.42 | 0.36 | >> | find-directory-files-recursively-5 (rpom 409600) | 0.91 | 0.25 | >> >> read_and_insert_process_output.diff (noticeable differences) >> >> | Function | gecko-dev | linux | >> | find-directory-files-recursively | 1.30 | 0.34 | >> | find-directory-files-recursively-5 | 1.03 | 0.25 | >> | find-directory-files-recursively (rpom 409600) | 1.20 | 0.35 | >> | find-directory-files-recursively-5 (rpom 409600) | (!!) 0.72 | 0.21 | >> >> So it seems like we have at least two potential ways to implement an >> asynchronous file listing routine that is as fast or faster than the >> synchronous one (if only thanks to starting the parsing in parallel). >> >> Combining the last patch together with using the very large value of >> read-process-output-max seems to yield the most benefit, but I'm not >> sure if it's appropriate to just raise that value in our code, though. >> >> Thoughts? > > I'm not sure what exactly is here to think about. Removing portions > of read_and_insert_process_output, or bypassing it entirely, is not > going to fly, because AFAIU it basically means we don't decode text, > which can only work with plain ASCII file names, and/or don't move the > markers in the process buffer, which also cannot be avoided. That one was really a test to see whether the extra handling added any meaningful consing to affect GC. Removing it didn't make a difference, table number 2, so no. > If you > want to conclude that inserting the process's output into a buffer > without consing Lisp strings is faster (which I'm not sure, see below, > but it could be true), That's what my tests seem to show, see table 3 (the last one). > then we could try extending > internal-default-process-filter (or writing a new filter function > similar to it) so that it inserts the stuff into the gap and then uses > decode_coding_gap, Can that work at all? By the time internal-default-process-filter is called, we have already turned the string from char* into Lisp_Object text, which we then pass to it. So consing has already happened, IIUC. > which converts inserted bytes in-place -- that, at > least, will be correct and will avoid consing intermediate temporary > strings from the process output, then decoding them, then inserting > them. Other than that, the -2 and -3 variants are very close > runners-up of -5, so maybe I'm missing something, but I see no reason > be too excited here? I mean, 0.89 vs 0.92? really? The important part is not 0.89 vs 0.92 (that would be meaningless indeed), but that we have an _asyncronous_ implementation of the feature that works as fast as the existing synchronous one (or faster! if we also bind read-process-output-max to a large value, the time is 0.72). The possible applications for that range from simple (printing progress bar while the scan is happening) to more advanced (launching a concurrent process where we pipe the received file names concurrently to 'xargs grep'), including visuals (xref buffer which shows the intermediate search results right away, updating them gradually, all without blocking the UI). > About inserting into the buffer: what we do is insert into the gap, > and when the gap becomes full, we enlarge it. Enlarging the gap > involves: (a) enlarging the chunk of memory allocated to buffer text > (which might mean we ask the OS for more memory), and (b) moving the > characters after the gap to the right to free space for inserting more > stuff. This is pretty fast, but still, with a large pipe buffer and a > lot of output, we do this many times, so it could add up to something > pretty tangible. It's hard to me to tell whether this is > significantly faster than consing strings and inserting them, only > measurements can tell. See the benchmark tables and the POC patch in my previous email. Using a better filter function would be ideal, but it seems like that's not going to fit the current design. Happy to be proven wrong, though.