From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Ihor Radchenko Newsgroups: gmane.emacs.bugs Subject: bug#64735: 29.0.92; find invocations are ~15x slower because of ignores Date: Thu, 20 Jul 2023 15:42:17 +0000 Message-ID: <87tttypp2e.fsf@localhost> References: <1fd5e3ed-e1c3-5d6e-897f-1d5d55e379fa@gutov.dev> <87wmyupvlw.fsf@localhost> <5c4d9bea-3eb9-b262-138a-4ea0cb203436@gutov.dev> Mime-Version: 1.0 Content-Type: text/plain Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="13785"; mail-complaints-to="usenet@ciao.gmane.io" Cc: Spencer Baugh , 64735@debbugs.gnu.org To: Dmitry Gutov Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Thu Jul 20 17:43:22 2023 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1qMVoI-0003Px-9S for geb-bug-gnu-emacs@m.gmane-mx.org; Thu, 20 Jul 2023 17:43:22 +0200 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qMVnz-0004nu-IR; Thu, 20 Jul 2023 11:43:03 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qMVny-0004nR-9j for bug-gnu-emacs@gnu.org; Thu, 20 Jul 2023 11:43:02 -0400 Original-Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1qMVny-0008Jl-2C for bug-gnu-emacs@gnu.org; Thu, 20 Jul 2023 11:43:02 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1qMVnx-0005ei-UA for bug-gnu-emacs@gnu.org; Thu, 20 Jul 2023 11:43:01 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Ihor Radchenko Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Thu, 20 Jul 2023 15:43:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 64735 X-GNU-PR-Package: emacs Original-Received: via spool by 64735-submit@debbugs.gnu.org id=B64735.168986774421695 (code B ref 64735); Thu, 20 Jul 2023 15:43:01 +0000 Original-Received: (at 64735) by debbugs.gnu.org; 20 Jul 2023 15:42:24 +0000 Original-Received: from localhost ([127.0.0.1]:59487 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qMVnL-0005dq-Uh for submit@debbugs.gnu.org; Thu, 20 Jul 2023 11:42:24 -0400 Original-Received: from mout02.posteo.de ([185.67.36.66]:51251) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1qMVnI-0005db-8R for 64735@debbugs.gnu.org; Thu, 20 Jul 2023 11:42:21 -0400 Original-Received: from submission (posteo.de [185.67.36.169]) by mout02.posteo.de (Postfix) with ESMTPS id D68B3240101 for <64735@debbugs.gnu.org>; Thu, 20 Jul 2023 17:42:13 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017; t=1689867733; bh=YZc7W2lILHnFsPNcNeE1M++pUIcYef2prBgPrynZ/3c=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:From; b=YfE3crK8tyMvo+wbAFsbjnq9B0Jtmzm+RDsFyfwQQ4XxpWBZBzSU9ApPYaN5X0ph6 F0dXtEa3zS0cyQv7F2tTI3gRYFKCrhR6hnPhjfK8jAgkLWrmMMVLNsZbEMjIfhE2XI gygfT557H3TALI9F5tQDn3xcXP2kjHW80g/+ZnWoL/xjaHTHEEK8JzNWtuLQobKcdG 8AmFLy2xMS6jmY36Q420L9YnG9NrMlxjKuM+2kXgvNXHZwoRkR39G6sgiKfxkOtZzB pKL3xkMqOwYms7K2ZvRxHBC4H/LxNsJJjtEwPulXje2eFb5q5OYbXhq0/lwLBO7Gri BbLvUUbfsHtBw== Original-Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4R6H4K2HtCz9rxW; Thu, 20 Jul 2023 17:42:09 +0200 (CEST) In-Reply-To: <5c4d9bea-3eb9-b262-138a-4ea0cb203436@gutov.dev> X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.bugs:265606 Archived-At: Dmitry Gutov writes: >>> ... Last I checked, Lisp-native file >>> listing was simply slower than 'find'. >> >> Could it be changed? >> In my tests, I was able to improve performance of the built-in >> `directory-files-recursively' simply by disabling >> `file-name-handler-alist' around its call. > > Then it won't work with Tramp, right? I think it's pretty nifty that > project-find-regexp and dired-do-find-regexp work over Tramp. Sure. It might also be optimized. Without trying to convince find devs to do something about regexp handling. And things are not as horrible as 15x slowdown in find. >> See https://yhetil.org/emacs-devel/87cz0p2xlc.fsf@localhost/ >> (the thread also continues off-list, and it looks like there is a lot of >> room for improvement in this area) > > Does it get close enough to the performance of 'find' this way? Comparable: (ignore (let ((gc-cons-threshold most-positive-fixnum)) (benchmark-progn (directory-files-recursively "/home/yantar92/.data" "")))) ;; Elapsed time: 0.633713s (ignore (let ((gc-cons-threshold most-positive-fixnum)) (benchmark-progn (let ((file-name-handler-alist)) (directory-files-recursively "/home/yantar92/.data" ""))))) ;; Elapsed time: 0.324341s ;; time find /home/yantar92/.data >/dev/null ;; real 0m0.129s ;; user 0m0.017s ;; sys 0m0.111s > Also note that processing all matches in Lisp, with many ignores > entries, will incur the proportional overhead in Lisp. Which might be > relatively slow as well. Not significant. I tried to unwrap recursion in `directory-files-recursively' and tried to play around with regexp matching of the file list itself - no significant impact compared to `file-name-handler-alist'. I am pretty sure that Emacs's native file routines can be optimized to the level of find. -- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at . Support Org development at , or support my work at