From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Dmitry Gutov Newsgroups: gmane.emacs.bugs Subject: bug#73320: [PATCH] project--vc-list-files: use Git's sparse-index Date: Thu, 19 Sep 2024 01:27:03 +0300 Message-ID: <73758f39-1e18-471a-9dfb-0ceade12dacf@gutov.dev> References: Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="2100"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Mozilla Thunderbird Cc: 73320@debbugs.gnu.org To: Sean Allred Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Thu Sep 19 00:28:07 2024 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1sr39a-0000N5-AY for geb-bug-gnu-emacs@m.gmane-mx.org; Thu, 19 Sep 2024 00:28:07 +0200 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sr39H-0001i4-2w; Wed, 18 Sep 2024 18:27:47 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sr39F-0001hW-Gj for bug-gnu-emacs@gnu.org; Wed, 18 Sep 2024 18:27:45 -0400 Original-Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1sr39F-0001ii-7R for bug-gnu-emacs@gnu.org; Wed, 18 Sep 2024 18:27:45 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debbugs.gnu.org; s=debbugs-gnu-org; h=In-Reply-To:From:References:MIME-Version:Date:To:Subject; bh=gEl2kfhz4a30d2JO4FbO9y6aKk6bSUzDpeRwyQi3X3E=; b=Qj6Gt2z1zxnCeDrlf5MDdhg2fCgmEQfW11oCjmcNGrawjfZ/ht6ucJOu0yWKdFDmu2ztHq1Bw+rTbaJYNDeU9P7I4gcwW81DXIENzpRh7AGHodTdO4FkROjQ5kpszbapRh6B7BIr4D8rBiq12vXTexUPSLKluI/YHFjS6qLlhwH+JjTcI+NnYlMRQVKjuOZhq29c5FggtTPjBC4MjbDiupQ/TnXdThTIcfA8QQhw4kYFTGOD+6hgTeosjjDHfGSP8ugDRZ3G7DVYJvN45A/oypEt9AKbikDHYBjUCY8ubv/AGCz3iwMrZbbVP5BBCMTtKR/lUrBQfsmwv6976RosDQ==; Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1sr39V-0000Lj-PC for bug-gnu-emacs@gnu.org; Wed, 18 Sep 2024 18:28:01 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Dmitry Gutov Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Wed, 18 Sep 2024 22:28:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 73320 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: patch Original-Received: via spool by 73320-submit@debbugs.gnu.org id=B73320.17266984521305 (code B ref 73320); Wed, 18 Sep 2024 22:28:01 +0000 Original-Received: (at 73320) by debbugs.gnu.org; 18 Sep 2024 22:27:32 +0000 Original-Received: from localhost ([127.0.0.1]:59361 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sr391-0000Ky-LO for submit@debbugs.gnu.org; Wed, 18 Sep 2024 18:27:32 -0400 Original-Received: from fhigh6-smtp.messagingengine.com ([103.168.172.157]:58025) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sr38z-0000Kd-36 for 73320@debbugs.gnu.org; Wed, 18 Sep 2024 18:27:30 -0400 Original-Received: from phl-compute-11.internal (phl-compute-11.phl.internal [10.202.2.51]) by mailfhigh.phl.internal (Postfix) with ESMTP id C97511140122; Wed, 18 Sep 2024 18:27:06 -0400 (EDT) Original-Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-11.internal (MEProxy); Wed, 18 Sep 2024 18:27:06 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gutov.dev; h=cc :cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to; s=fm3; t=1726698426; x=1726784826; bh=gEl2kfhz4a30d2JO4FbO9y6aKk6bSUzDpeRwyQi3X3E=; b= ePpfDalQIKxWEihW5b3X6BEvwVdhhMLj6p4tMebjqc6UruBIlZ4GKcow6tOgvw3D dXfoHH0phpykqFzfHmb1e9rfwKOe+dt6X8YJ2TcMKd/7SEF+eSlcec2Z19mOKjBp i98EEGt28uwzB+roJwncQIOqh0y9llIvUWur7U5NrwUtqCBOxpm9F64SkepmO1DN 887ACq/LoENxbwyyp7IfPLvR0rLvRydLHNGx0XyJVklIgkhOK7CeypqskmeA8mhT nX9zG7uMwtj3tVF86DSJeRP3/kiBDAzJeeYB9WA0wiNHczlDD+t8SHtIYqV7Cv9z 7+qj3tywKIZLkRZPoF1TMQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t=1726698426; x= 1726784826; bh=gEl2kfhz4a30d2JO4FbO9y6aKk6bSUzDpeRwyQi3X3E=; b=Y EVpwBhtfK9IC13P7ObEraCFM4iU3hMF+yQVkDF3wx7xyUFKCLldch9tsb4+Rw+o6 xuFWcniI1E2LBi0i515cpMaJ07r7IvZw4+Cj8pA6iiZH4yH1UW/wwX//j/Klcg4w NhoVLUHbWRzt9cf/HhYMLweNlm4N0sz3v0iYOQO6Iyt/ZfJwkjqcDUOIUskHf9iH Q4GUEC7PkvffafyRjLT5FCucuOk2UvcR5W+8Xm+xTQMigrBjF3TqqcjrnlX2tbmx 5ZwISW51bzKQyqj5NFCQeo9e4aq591SyWSNwH8QvdgWvUAXjnlY+6/ROFydXK0oJ lwHv3Jv/W8hK90d4YvZ0w== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeftddrudeltddgudduucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdggtfgfnhhsuhgsshgtrhhisggvpdfu rfetoffkrfgpnffqhgenuceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnh htshculddquddttddmnecujfgurhepkfffgggfuffvvehfhfgjtgfgsehtjeertddtvdej necuhfhrohhmpeffmhhithhrhicuifhuthhovhcuoegumhhithhrhiesghhuthhovhdrug gvvheqnecuggftrfgrthhtvghrnhepteduleejgeehtefgheegjeekueehvdevieekueef tddvtdevfefhvdevgedujeehnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpe hmrghilhhfrhhomhepughmihhtrhihsehguhhtohhvrdguvghvpdhnsggprhgtphhtthho pedvpdhmohguvgepshhmthhpohhuthdprhgtphhtthhopegrlhhlrhgvugdrshgvrghnse hgmhgrihhlrdgtohhmpdhrtghpthhtohepjeeffedvtdesuggvsggsuhhgshdrghhnuhdr ohhrgh X-ME-Proxy: Feedback-ID: i07de48aa:Fastmail Original-Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 18 Sep 2024 18:27:05 -0400 (EDT) Content-Language: en-US In-Reply-To: X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.bugs:292018 Archived-At: On 18/09/2024 03:36, Sean Allred wrote: > Dmitry Gutov writes: >> The submitted patch has a comma inside which results in an error > > Well that's embarrassing. I've resolved this in my branch. I'm not sure > why I wasn't seeing an issue locally; perhaps I made this error somehow > after I eval'd the form for testing. Thanks for catching! No problem! >> But let's start from the beginning. Could you help me set up a sparse >> repo that would help test out the change? >> >> I took a large-ish checkout with shallow history and set up the sparse >> config like this: >> >> git sparse-checkout init --cone >> git sparse-checkout set gfx media >> >> With that, both 'git status' and 'ls' behave as expected - no extra >> directories around (just the top-level files and two subdirs). >> >> But 'git ls-files' and 'git ls-files --sparse' continue to show the >> same large output. Any idea what could be wrong? My version of Git, >> perhaps? Which is 2.40.1. > > Sure! You're very close. The key is here in git-ls-files(1): > > --sparse > If the index is sparse, show the sparse directories without > expanding to the contained files. Sparse directories will be shown > with a trailing slash, such as "x/" for a sparse directory "x". > > combined with this in git-sparse-checkout(1): > > Use the --[no-]sparse-index option to use a sparse index (the > default is to not use it). > > I fell into this too when setting up my worktree for testing; normally > our internal tooling takes care of this for me. To resolve this issue in > an existing checkout, you can simply > > git sparse-checkout init --sparse-index > > and that will rewrite your index file using sparse-index rules. Thanks! So the key here is using "sparse index", not just "sparse checkout", since these are two different features. Which have to be enabled separately for sort of "gradual rollout", I guess. >> Yeah, I expect project-find-regexp, project-search, >> project-query-replace-regexp might start misbehaving without >> additional filtering -- either throwing up errors or, best case, >> continuing to search through the "hidden" directories. > > Not sure how best to track that we should come back to this, but yeah. > It seems like the right place to add some sort of switch would be in the > `project-files` defmethod. From here, it looks like all the functions > you mention could choose the behavior right for them. (Based on the > function names alone -- it seems they would /also/ be interested in > operating on only those files which exist on disk.) I think we can just remove the names ending with '/'. The built-in commands don't seem to error out on them right now - probably because there is some protection against nonexistent files - but those files are (were) still shown as completions for project-find-file. Try out this addition please. The performance here seems about the same even with a large list (something I was worried about): diff --git a/lisp/progmodes/project.el b/lisp/progmodes/project.el index b29d5ed5404..a2e3f3f52e6 100644 --- a/lisp/progmodes/project.el +++ b/lisp/progmodes/project.el @@ -663,7 +663,7 @@ project--vc-list-files (pcase backend (`Git (let* ((default-directory (expand-file-name (file-name-as-directory dir))) - (args '("-z")) + (args '("-z" "--sparse")) (vc-git-use-literal-pathspecs nil) (include-untracked (project--value-in-dir 'project-vc-include-untracked @@ -703,7 +703,8 @@ project--vc-list-files (delq nil (mapcar (lambda (file) - (unless (member file submodules) + (unless (or (member file submodules) + (eq ?/ (aref file (1- (length file))))) (if project-files-relative-names file (concat default-directory file)))) > Incidentally looking at the version check within `project-files`, it's > worthwhile to point out that `--sparse` is likely /not/ compatible with > ancient versions of Git. Does vc have any sort of policy on requiring > recent versions of these tools? If the answer is 'not really', I'll > additionally want to add some sort of protection against using > `--sparse` when the Git version won't understand it. This should be easy > enough to do within the implementation of `project--vc-list-files`. IIRC it was something like "should work on the CentOS stable", and maybe CentOS N-1 as well. But the release-based distro was discontinued since the last time this question came up ;-( We can call vc-git--program-version the same way it's used in vc-git-state. Which version should we make the minimum?