From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Dmitry Gutov Newsgroups: gmane.emacs.bugs Subject: bug#69233: 30.0.50; project-files + project-find-file is slow in large repositories Date: Sun, 5 May 2024 06:32:12 +0300 Message-ID: References: <1b566e9e-eca5-4746-8e31-4155d35ce7a8@gutov.dev> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="36530"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Mozilla Thunderbird Cc: 69233@debbugs.gnu.org, 69188@debbugs.gnu.org To: Spencer Baugh Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Sun May 05 05:32:50 2024 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1s3ScL-0009E2-4u for geb-bug-gnu-emacs@m.gmane-mx.org; Sun, 05 May 2024 05:32:49 +0200 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1s3ScE-0008Q1-2S; Sat, 04 May 2024 23:32:42 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1s3ScB-0008P1-Jh for bug-gnu-emacs@gnu.org; Sat, 04 May 2024 23:32:39 -0400 Original-Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1s3ScB-0007eN-8O for bug-gnu-emacs@gnu.org; Sat, 04 May 2024 23:32:39 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1s3ScY-0007Ez-JO for bug-gnu-emacs@gnu.org; Sat, 04 May 2024 23:33:02 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Dmitry Gutov Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Sun, 05 May 2024 03:33:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 69233 X-GNU-PR-Package: emacs Original-Received: via spool by 69233-submit@debbugs.gnu.org id=B69233.171487997427814 (code B ref 69233); Sun, 05 May 2024 03:33:02 +0000 Original-Received: (at 69233) by debbugs.gnu.org; 5 May 2024 03:32:54 +0000 Original-Received: from localhost ([127.0.0.1]:57291 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s3ScP-0007ET-Pe for submit@debbugs.gnu.org; Sat, 04 May 2024 23:32:54 -0400 Original-Received: from wfhigh2-smtp.messagingengine.com ([64.147.123.153]:39079) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1s3ScK-0007EE-AP; Sat, 04 May 2024 23:32:51 -0400 Original-Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailfhigh.west.internal (Postfix) with ESMTP id CD3C718000E4; Sat, 4 May 2024 23:32:17 -0400 (EDT) Original-Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Sat, 04 May 2024 23:32:18 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc :cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to; s=fm2; t=1714879937; x=1714966337; bh=u6UhfW3JE9QRybsw3/JTjHmGPyhf2AnYrTBLf60RwhU=; b= bUHR7LsVOeI1SRgdJ/tEuPlfnQaVrK0DdEEVA57O2j3OAeYvrXo/iHI2HddD+EFR FWohgveFBOgrKfojHQbyufEDNFGnNK6bwYIf0Wy6JV32GoWhxZCnH7Qia/jk7dbk sRqW4mp8WZau7ZjcwWMlPLoo5g+Q0wKvEwuYz2Tir/MVUpY39Pgx+ZEPniK7Pxkw WTYoJXl/jVYw9g3xPYjqZWwo1yfq3B222MVIPsix84fJfIIY2Omw+1J1e1iAjHcD zkHb2yMXJhCHEIv94z4RwPh4uqxk7t222gF6by1M9ccnmBu0Sfo2zHd7/9hJM55W kvnPFRGdNf80MG5XqBdyCA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=1714879937; x= 1714966337; bh=u6UhfW3JE9QRybsw3/JTjHmGPyhf2AnYrTBLf60RwhU=; b=j AUQIwFv5iAFHgah7xLQC0xHlKl7WkEtT4vrdSRtGYBBjghryvypd7ZmQTOv3P+wB Iu8Rr3wE5OGYifAmbaioXP+D+CtdKKDMuCt4uk3pu5nh2IjcTvGG7iI/TfGk7KyB Ey7NfE8Qnzn+M2yD5R2SUW4V/GSVyHzpENos1c08ByGNJJGCkf6ca2H1Y+w88SK8 bcXOnOwLn06I4L+Qid3MeKa3bR5hKUZkohPj7jCXi2tj5V10MVhdfj/2PmgYEv01 mj0nRSk4EDjuN+gPZyHqxBrT8gBslxzF5H7r/cY8jfJvjmCmEHaGSyPnNF2ptdb1 U58rZmYr1cOgNK3RmW/ew== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvddvfedgjeefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepkfffgggfuffvvehfhfgjtgfgsehtjeertddtvdejnecuhfhrohhmpeffmhhi thhrhicuifhuthhovhcuoegumhhithhrhiesghhuthhovhdruggvvheqnecuggftrfgrth htvghrnhepteduleejgeehtefgheegjeekueehvdevieekueeftddvtdevfefhvdevgedu jeehnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepug hmihhtrhihsehguhhtohhvrdguvghv X-ME-Proxy: Feedback-ID: i0e71465a:Fastmail Original-Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sat, 4 May 2024 23:32:15 -0400 (EDT) Content-Language: en-US In-Reply-To: X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.bugs:284477 Archived-At: On 30/04/2024 00:04, Spencer Baugh wrote: > Oh, interesting, I see roughly the same result. > > Benchmarking with: > (benchmark-run 10 (project-files (project-current))) > > Running in my long-lived existing Emacs 29 session: > Old: > (4.434228319 14 2.850654906999921) > New: > (4.983809167 16 3.2989908669999295) > > In Emacs 29 emacs -Q: > Old: > (3.5112438729999997 130 1.9230644630000002) > New: > (3.819248509 171 2.309731412) > > But, in Emacs 30 emacs -Q: > Old: > (7.949549188 65 3.3445626799999992) > New: > (7.270785783999999 87 4.0610532379999995) > > So... the performance improvement seems highly unreliable. Probably not > worth changing this area, then - the other patch to allow relative files > will probably be more worth it. All right then, let's hold off on this potential change for now, and maybe revisit it later. Maybe the new GC engine will swing the needle in one or the other direction. > I think the defvar approach seems reasonable. > > The existing project-read-file-name-function certainly don't expect > relative names, but they do actually work OK. e.g. > > (project--read-file-cpd-relative "" '("foo/bar" "foo1/bar") nil 'minibuffer-history) Evaluating this one with the version in master results in Debugger entered--Lisp error: (wrong-type-argument stringp nil) expand-file-name(nil) hence the associated change in the patch. > (project--read-file-absolute "" '("foo/bar" "foo1/bar") nil 'minibuffer-history) No errors here, but two problems are that a) it doesn't show the default-directory [meaning no indication in which project the read is happening], and b) returning the relative name will mess up the file-name-history entry. Good thing you noted the latter, it needs explicit handling. The former can be be shown in the prompt, at least. > Both complete fine and return a filename fine. read-file-cpd-relative > returns an absolute filename, read-file-absolute reutrns a relative > filename. > > Maybe the same is true for any custom project-read-file-name-functions > that exist? Maybe they will just work? So, apparently not. Anyway, I've pushed the patch in commit 370b216f086. Here's hoping the breakage will be minimal. >>> However, that would make it easy for project-files as a whole to be >>> asynchronous. Then that would allow project-find-file to start the >>> listing in the background, and then we'd write a completion table which >>> completes only over whatever files we've already read into Emacs. I >>> think this would be a lot nicer for most use-cases, and I'd again be >>> happy to implement this. >> >> Could this be that simple? >> >> Whatever the source of the file listing, as soon as the UI (or >> completion styles) calls try-completion or all-completions, the search >> has to finish first, shouldn't it? That seems like the semantics of >> this API. Or if perhaps we allow it to operate on incomplete results, >> how would we indicate to the user at the end that the scan has >> finished, and they can press TAB once more to refresh the results? Or >> perhaps to be able to find a file they hadn't managed to find in the >> incomplete set. >> >> This seems like it might require both a new UI and an extension of >> completion table API. E.g. in certain cases we could say that we only >> need N matches, so if the current incomplete set can provide as many, >> we don't have to wait until the end. But 'try-completion' would become >> unreliable either way. > > Yes, that's all true, and this is definitely not the intended semantics > of the API, but I vaguely suspect it might be fine in practice? That > vague suspicion can wait until later, though, because I think the more > conservative approach you suggest is also a good improvement on its own. Some async stuff could make a big improvement on top of it, but it seems to require a fair bit more complexity. >> Even if keeping to the most conservative approach, though, it should >> be possible to at least render the prompt before the file listing is >> finished. That could make the UI look a bit more responsive. > > True, that would be pretty nice. And further I suppose in the case of > the default completion UI (which doesn't automatically display > completions), the user can even type some input before hitting TAB and > waiting. It could be advantageous if the search process starts right when (or before) the prompt is shown, then by the type the first input is entered the search could either be finished or have found some matches at least. > Also, I suppose that even non-default completion UIs would allow the > user to type input, if the non-default completion UI uses > while-no-input. So it would be a pretty responsive experience for such > UIs (assuming we are careful in our implementation and don't have bugs > when being interrupted). Not sure about this one: 1) If you only do the search while the user is not typing, it will finish later compared to the scheme in the previous paragraph. 2) Suppose you type a char, pause, then another one. Will the search start, abort, and then start again? That seems wasteful. I'd ultimately prefer a scheme where work isn't thrown away - but that would require a more complex API. Including a way to abort the background computation (since typing won't do that anymore). For some UIs and commands that makes sense (e.g. incremental interfaces like counsel-rg) because they perform the search with different inputs each time you type a new character. That kinds of works for small-to-medium projects, and you can enjoy the responsiveness of the process. I'm not sure about this approach for big projects.