From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Dmitry Gutov Newsgroups: gmane.emacs.bugs Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores Date: Mon, 11 Sep 2023 03:02:55 +0300 Message-ID: References: <83sf9f6wm0.fsf@gnu.org> <83sf9eub9d.fsf@gnu.org> <2d844a34-857d-3d59-b897-73372baac480@gutov.dev> <83bkg2tsu6.fsf@gnu.org> <83bd4246-ac41-90ec-1df3-02d0bd59ca44@gutov.dev> <834jlttv1p.fsf@gnu.org> <937c3b8e-7742-91b7-c2cf-4cadd0782f0c@gutov.dev> <83a5vlsanw.fsf@gnu.org> <69a98e2a-5816-d36b-9d04-8609291333cd@gutov.dev> <87351cs8no.fsf@localhost> <35163e56-607d-9c5b-e3e8-5d5b548b3cb7@gutov.dev> <878rb3m43b.fsf@localhost> <83v8e6lyi4.fsf@gnu.org> <35f8b664-0241-9f96-1aa0-20ca51b2d34c@gutov.dev> <59c30342-a7e0-d83b-a128-0faae4cbd633@gutov.dev> <83pm4bi6qa.fsf@gnu.org> <83bkfs2tw5.fsf@gnu.org> <18a0b4d8-32bd-3ecd-8db4-32608a1ebba7@gutov.dev> <83il8lxjcu.fsf@gnu.org> <2e21ec81-8e4f-4c02-ea15-43bd6da3daa7@gutov.dev> <8334zmtwwi.fsf@gnu.org> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="------------mo9wiY0Ph8bca03YUu6cB50g" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="30151"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.13.0 Cc: luangruo@yahoo.com, sbaugh@janestreet.com, yantar92@posteo.net, 64735@debbugs.gnu.org To: Eli Zaretskii Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Mon Sep 11 02:04:16 2023 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1qfUPX-0007a9-Bm for geb-bug-gnu-emacs@m.gmane-mx.org; Mon, 11 Sep 2023 02:04:16 +0200 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qfUPH-0000yr-Ls; Sun, 10 Sep 2023 20:03:59 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qfUPG-0000yi-7h for bug-gnu-emacs@gnu.org; Sun, 10 Sep 2023 20:03:58 -0400 Original-Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1qfUPF-0003Pd-Vr for bug-gnu-emacs@gnu.org; Sun, 10 Sep 2023 20:03:57 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1qfUPJ-0002J1-MG for bug-gnu-emacs@gnu.org; Sun, 10 Sep 2023 20:04:01 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Dmitry Gutov Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Mon, 11 Sep 2023 00:04:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 64735 X-GNU-PR-Package: emacs Original-Received: via spool by 64735-submit@debbugs.gnu.org id=B64735.16943905938808 (code B ref 64735); Mon, 11 Sep 2023 00:04:01 +0000 Original-Received: (at 64735) by debbugs.gnu.org; 11 Sep 2023 00:03:13 +0000 Original-Received: from localhost ([127.0.0.1]:51546 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qfUOW-0002Hz-L2 for submit@debbugs.gnu.org; Sun, 10 Sep 2023 20:03:13 -0400 Original-Received: from out3-smtp.messagingengine.com ([66.111.4.27]:42585) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qfUOR-0002Hj-LQ for 64735@debbugs.gnu.org; Sun, 10 Sep 2023 20:03:10 -0400 Original-Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailout.nyi.internal (Postfix) with ESMTP id 801EA5C0067; Sun, 10 Sep 2023 20:02:58 -0400 (EDT) Original-Received: from mailfrontend2 ([10.202.2.163]) by compute6.internal (MEProxy); Sun, 10 Sep 2023 20:02:58 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc :cc:content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1694390578; x=1694476978; bh=fL aeDQZpvI2i8RppxU+bKtKe1kdbtfqOkUEZdEflxMk=; b=rwqJdtXVmdI+kt+I+H 7FZ2dGe68GkUjA5uywEoXFwC2T/skhew4EuuV77lH7JWwDVNlkBI40uLMCiwsJGo hWz/iqLFaopwgaZxdAnnuuUezEBZMJtn8oUN0qMOd3RhgXHlw5xtv95MHnC8CqlE RSVLvp7VahwfpHK+g5zh+ZxlZXXqi9r3u41OB8yrnQnQy3IiLqelQUGZ0shBjo3p IwnjBxhLNEcc2+oWhWCu5l/4e4hvKD/InX2BlBpdgT+/y/alzu6J8vZ6kbKQUAmC IXi9RZA7zDa6R5Fax7CY2lRVDKUBYQQCJdPPt3ZICwaC1+SYYrFyyy5axNZEWT6t Ga0g== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm1; t=1694390578; x=1694476978; bh=fLaeDQZpvI2i8 RppxU+bKtKe1kdbtfqOkUEZdEflxMk=; b=JSys0Nq1YX2/n65YFOHX3m4xl3iQC dT0AbL3ZKkfOpZSoJ64YyayLCHM3Z3InFlh9Pp7uOjtZVU0//4i/3f+cchL1JsoJ 1sf8YKdLn0fphwWkPK78oiY5YbH16K3dxeaj4AtlqtZYg695hjUym1VR+WjTwM0g q0kvurBHieI/K2pa1SYuSvZyLwDsK9EV1PcUv14r71XK0Gwfx7cpE+8b/6dseMoH UAkyEy5nsW/VxhWM2uEqvcW/iabH+ndXukVqXGoq9Ph7ULlXKGHuUDlSFxLm2HoQ BUCVUjVg3xHCURJH8Th0zcO8gM4yHIxBug0mOb2n6RwdtBlqPv8f7fc/g== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrudeifedgvdelucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurheptgfkffggfgfuvfevfhfhjgesmhdtreertdefjeenucfhrhhomhepffhmihht rhihucfiuhhtohhvuceoughmihhtrhihsehguhhtohhvrdguvghvqeenucggtffrrghtth gvrhhnpeehteekgfetieeujeeuvddtvdelteffleejteduvdefffejieehheeuteffveei jeenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpegumh hithhrhiesghhuthhovhdruggvvh X-ME-Proxy: Feedback-ID: i0e71465a:Fastmail Original-Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sun, 10 Sep 2023 20:02:56 -0400 (EDT) Content-Language: en-US In-Reply-To: <8334zmtwwi.fsf@gnu.org> X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.bugs:270013 Archived-At: This is a multi-part message in MIME format. --------------mo9wiY0Ph8bca03YUu6cB50g Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit On 10/09/2023 08:33, Eli Zaretskii wrote: >> Date: Sun, 10 Sep 2023 04:30:24 +0300 >> Cc: luangruo@yahoo.com, sbaugh@janestreet.com, yantar92@posteo.net, >> 64735@debbugs.gnu.org >> From: Dmitry Gutov >> >> On 08/09/2023 09:35, Eli Zaretskii wrote: >>> I think you'd need to expose consing_until_gc to Lisp, and then you >>> can collect the data from Lisp. >> >> I can expose it to Lisp and print all three from post-gc-hook, but the >> result just looks like this: >> >> gc-pct 0.1 gc-thr 800000 cugc 4611686018427387903 >> >> Perhaps I need to add a hook which runs at the beginning of GC? Or of >> maybe_gc even? > > You could record its value in a local variable at the entry to > garbage_collect, and the expose that value to Lisp. That also doesn't seem to give much, given that the condition for entering 'maybe_garbage_collect' is (consing_until_gc < 0). I.e. we wait until it's down to 0, then garbage-collect. What we could perhaps do is add another hook (or a printer) at the beginning of maybe_gc, but either would result in lots and lots of output. >> Alternatively, (memory-use-counts) seems to retain some counters which >> don't get erased during garbage collection. > > Maybe using those will be good enough, indeed. I added this instrumentation: (defvar last-mem-counts '(0 0 0 0 0 0 0)) (defun gc-record-after () (let* ((counts (memory-use-counts)) (diff (cl-map 'list (lambda (old new) (- new old)) last-mem-counts counts))) (setq last-mem-counts counts) (message "counts diff %s" diff))) (add-hook 'post-gc-hook #'gc-record-after) so that after each garbage collection we print the differences in all counters (CONSES FLOATS VECTOR-CELLS SYMBOLS STRING-CHARS INTERVALS STRINGS). And a message call when the process finishes. And made those recordings during the benchmark runs of two different listing methods (one using make-process, another using process-file) to list all files in a large directory (there are ~200000 files there). The make-process one I also ran with a different (large) value of read-process-output-max. Results attached. What's in there? First of all, for find-directory-files-recursively-3, there are 0 garbage collections between the beginning of the function and when we start parsing the output (no GCs while the process is writing to the buffer synchronously). I guess inserting output in a buffer doesn't increase consing, so there's nothing to GC? Next: for find-directory-files-recursively-2, the process only finishes at the end, when all GC cycles are done for. I suppose that also means we block the process's output while Lisp is running, and also that whatever GC events occur might coincide with the chunks of output coming from the process, and however many of them turn out to be in total. So there is also a second recording for find-directory-files-recursively-2 with read-process-output-max=409600. It does improve the performance significantly (and reduce the number of GC pauses). I guess what I'm still not clear on, is whether the number of GC pauses is fewer because of less consing (the only column that looks significantly different is the 3rd: VECTOR-CELLS), or because the process finishes faster due to larger buffers, which itself causes fewer calls to maybe_gc. And, of course, what else could be done to reduce the time spent in GC in the asynchronous case. --------------mo9wiY0Ph8bca03YUu6cB50g Content-Type: text/plain; charset=UTF-8; name="gcs2.txt" Content-Disposition: attachment; filename="gcs2.txt" Content-Transfer-Encoding: base64 ZmluZC1kaXJlY3RvcnktZmlsZXMtcmVjdXJzaXZlbHktMgoKVXNlcyBtYWtlLXByb2Nlc3Mg YW5kIDpmaWx0ZXIgdG8gcGFyc2UgdGhlIG91dHB1dApjb25jdXJyZW50bHkgd2l0aCB0aGUg cHJvY2Vzcy4KCldpdGggKHJlYWQtcHJvY2Vzcy1vdXRwdXQtbWF4IDQwOTYpOgoKc3RhcnQg bm93CmNvdW50cyBkaWZmICg3NTg0MCAxMyAzMTE3NyA2MCAzNDM0NDMgMzQ5NiA0NzQ4KQpj b3VudHMgZGlmZiAoNDE5NDYgMSA0NjAgMCAxMjI2NDk0IDAgODQyNSkKY291bnRzIGRpZmYg KDQzMTY1IDEgNDUwIDAgMTI4NDIxNCAwIDg5NTEpCmNvdW50cyBkaWZmICg0MzUxMyAxIDM2 NCAwIDEzNDMzMTYgMCAxMDEyNSkKY291bnRzIGRpZmYgKDQzMjAwIDEgMzg0IDAgMTQ3OTA0 OCAwIDk3NjYpCmNvdW50cyBkaWZmICg0NjIyMCAxIDQyOCAwIDE1Mjg4NjMgMCAxMDI0MikK Y291bnRzIGRpZmYgKDQzMTI1IDEgNDYyIDAgMTc2NzA2OCAwIDg3OTApCmNvdW50cyBkaWZm ICg0OTExOCAxIDQ1OCAwIDE3MjMyNzEgMCAxMDgzMikKY291bnRzIGRpZmYgKDUzMTU2IDEg NTcyIDAgMTc4OTkxOSAwIDEwNzc0KQpjb3VudHMgZGlmZiAoNTc3NTUgMSA1NDggMCAxNzgz Mjg2IDAgMTI2MDApCmNvdW50cyBkaWZmICg2MjE3MSAxIDU1NCAwIDE3OTUyMTYgMCAxMzk5 NSkKY291bnRzIGRpZmYgKDYyMDIwIDEgNTUwIDAgMTk2MzI1NSAwIDEzOTk2KQpjb3VudHMg ZGlmZiAoNTQ1NTkgMSA2MTYgMCAyMzg3MzA4IDAgMTA3MDApCmNvdW50cyBkaWZmICg1NjQy OCAxIDYzNCAwIDI1MTMyMTkgMCAxMTA5NSkKY291bnRzIGRpZmYgKDYyNjExIDEgNjU4IDAg MjUxMDc1NiAwIDEyODY0KQpjb3VudHMgZGlmZiAoNjc1NjAgMSA3MDggMCAyNTc0MzEyIDAg MTM4OTkpCmNvdW50cyBkaWZmICg3ODE1NCAxIDkyOCAwIDI1NzIyNzMgMCAxNDcxNCkKY291 bnRzIGRpZmYgKDg2Nzk0IDEgOTc2IDAgMjUyMDkxNSAwIDE3MDA0KQpjb3VudHMgZGlmZiAo NzgxMTIgMSA4NzQgMCAyOTQzNTQ4IDAgMTUzNjcpCmNvdW50cyBkaWZmICg3OTQ0MyAxIDg5 NCAwIDMxMzg5NDggMCAxNTU1OSkKY291bnRzIGRpZmYgKDgxODYxIDEgOTg0IDAgMzM0Mzc2 NCAwIDE1MjYwKQpjb3VudHMgZGlmZiAoODc3MjQgMSAxMDMwIDAgMzQzMDk2OSAwIDE2NjUw KQpjb3VudHMgZGlmZiAoODg1MzIgMSA5MDIgMCAzNTkxMDUyIDAgMTg0ODcpCmNvdW50cyBk aWZmICg5MjA4MyAxIDk1MiAwIDM3NjkyOTAgMCAxOTA2NSkKPGZpbmlzaGVkXG4+CkVsYXBz ZWQgdGltZTogMS4zNDQ0MjJzICgwLjc0NzEyNnMgaW4gMjQgR0NzKQoKQW5kIGhlcmUncyB3 aXRoIChyZWFkLXByb2Nlc3Mtb3V0cHV0LW1heCA0MDk2MDApOgoKc3RhcnQgbm93CmNvdW50 cyBkaWZmICg1Nzk2NyAxIDQwNDAgMSA5ODE5MTIgMTA2IDc3MzEpCmNvdW50cyBkaWZmICgz MjA3NSAxIDIwIDAgMTkxOTA5NiAwIDEwNTYwKQpjb3VudHMgZGlmZiAoNDM0MzEgMSAxOCAw IDIyNTkzMTQgMCAxNDM3MSkKY291bnRzIGRpZmYgKDQ2MzM1IDEgMTggMCAyNDI2MjkwIDAg MTUzMzkpCmNvdW50cyBkaWZmICgzMTg3MiAxIDE4IDAgMjQ0NzYzOSAwIDEwNTE4KQpjb3Vu dHMgZGlmZiAoNDY1MjcgMSAxOCAwIDIzMjgwNDIgMCAxNTQwMykKY291bnRzIGRpZmYgKDQy NDY4IDEgMTggMCAyMDk5OTc2IDAgMTQwNTApCmNvdW50cyBkaWZmICg0ODY0OCAxIDE4IDAg MjMwMjcxMyAwIDE2MTEwKQpjb3VudHMgZGlmZiAoNTA0MDQgMSAyMCAwIDMyNjA5MjEgMCAx NjY2OSkKY291bnRzIGRpZmYgKDQwMTQ3IDEgMjAgMCAzMjY0NDYzIDAgMTMyNTEpCmNvdW50 cyBkaWZmICg0ODExOCAxIDIwIDAgMzI2MTcyNSAwIDE1OTA4KQpjb3VudHMgZGlmZiAoNjA3 MzIgMSAyODIgMCAyNzkxMDAzIDAgMTY3ODUpCmNvdW50cyBkaWZmICg3MTMyOSAxIDUwNiAw IDI3NjIyMzcgMCAxNzQ4NykKY291bnRzIGRpZmYgKDYxNDU1IDEgMzQyIDAgMzE5Mjc3MSAw IDE2MjcxKQpjb3VudHMgZGlmZiAoNDkwMzUgMSAzMCAwIDM2NjM3MTUgMCAxNjA4NSkKY291 bnRzIGRpZmYgKDU4NjUxIDEgMjM2IDAgMzc4Mzg4OCAwIDE2NjgzKQpjb3VudHMgZGlmZiAo NTcxMzIgMSAyNCAwIDQ1NTc2ODggMCAxODg2MikKY291bnRzIGRpZmYgKDcxMzE5IDEgMjQg MCA0NzY5ODkxIDAgMjM1OTEpCjxmaW5pc2hlZFxuPgpFbGFwc2VkIHRpbWU6IDAuODkwNzEw cyAoMC41NDY0ODZzIGluIDE4IEdDcykK --------------mo9wiY0Ph8bca03YUu6cB50g Content-Type: text/plain; charset=UTF-8; name="gcs3.txt" Content-Disposition: attachment; filename="gcs3.txt" Content-Transfer-Encoding: base64 ZmluZC1kaXJlY3RvcnktZmlsZXMtcmVjdXJzaXZlbHktMwoKVXNlcyBwcm9jZXNzLWZpbGUs IHBhcnNlcyB0aGUgYnVmZmVyCndpdGggc2VhcmNoLWZvcndhcmQgYXQgdGhlIGVuZC4KCnN0 YXJ0IG5vdwo8cHJvY2VzcyBmaW5pc2hlZCwgbm93IHBhcnNpbmc+CmNvdW50cyBkaWZmICg2 Mjc3MSA1IDI2NjI5IDYzIDQ1ODIxMSAzMjIzIDgwMzgpCmNvdW50cyBkaWZmICgxNzA0NSAx IDEyIDAgMTI4ODE1MyAwIDE2OTQ5KQpjb3VudHMgZGlmZiAoMTgzMDEgMSAxMiAwIDE0MzIx NjUgMCAxODIwNSkKY291bnRzIGRpZmYgKDE3NjQzIDEgMTIgMCAxNzE2Mjk0IDAgMTc1NDcp CmNvdW50cyBkaWZmICgyMTkxNyAxIDEyIDAgMTcyNjQ2MiAwIDIxODIxKQpjb3VudHMgZGlm ZiAoMjU4ODggMSAxMiAwIDE3NzczNzEgMCAyNTc5MikKY291bnRzIGRpZmYgKDIxNzQzIDEg MTIgMCAyMzQ1MTQzIDAgMjE2NDcpCmNvdW50cyBkaWZmICgyNDAzNSAxIDEyIDAgMjU2MTQ5 MSAwIDIzOTM5KQpjb3VudHMgZGlmZiAoMzAwMjggMSAxMiAwIDI1OTMwNjkgMCAyOTkzMikK Y291bnRzIGRpZmYgKDI5NjI3IDEgMTIgMCAzMDQxMzA3IDAgMjk1MzEpCmNvdW50cyBkaWZm ICgzMDE0MCAxIDEyIDAgMzQ3OTIwOSAwIDMwMDQ0KQpjb3VudHMgZGlmZiAoMzUxODEgMSAx MiAwIDM2OTA0ODAgMCAzNTA4NSkKRWxhcHNlZCB0aW1lOiAwLjk0MzA5MHMgKDAuMzUxNzk5 cyBpbiAxMiBHQ3MpCg== --------------mo9wiY0Ph8bca03YUu6cB50g--