From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Sean Allred Newsgroups: gmane.emacs.bugs Subject: bug#73320: [PATCH] project--vc-list-files: use Git's sparse-index Date: Wed, 18 Sep 2024 23:25:57 -0500 Message-ID: References: <73758f39-1e18-471a-9dfb-0ceade12dacf@gutov.dev> Mime-Version: 1.0 Content-Type: text/plain Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="14121"; mail-complaints-to="usenet@ciao.gmane.io" Cc: 73320@debbugs.gnu.org To: Dmitry Gutov Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Thu Sep 19 06:28:06 2024 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1sr8lx-0003Rv-Ea for geb-bug-gnu-emacs@m.gmane-mx.org; Thu, 19 Sep 2024 06:28:05 +0200 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sr8le-00004t-J5; Thu, 19 Sep 2024 00:27:46 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sr8ld-0008WP-92 for bug-gnu-emacs@gnu.org; Thu, 19 Sep 2024 00:27:45 -0400 Original-Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1sr8lc-00059U-W0 for bug-gnu-emacs@gnu.org; Thu, 19 Sep 2024 00:27:45 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debbugs.gnu.org; s=debbugs-gnu-org; h=MIME-Version:Date:References:In-Reply-To:From:To:Subject; bh=Jc6wqY66udr+WSJ8DzWKxfIKPwdF0DpIzspM25WBpzI=; b=befgEoEYIyvhFGG/BW+zlhnI7GdM0YNEhJjbvXJePmRb3q2HhsuYMEa5tZFoE0HjpAEkxp/Pf8ZtlHXVHSXAhuuw6ggHFknywWCdj+MWVJc4QcqWTqig0gD8rasjueAXob2vuBW7Tr0N8RtJ7T3mK7pTqDOUpVm7MCSnDQUrAOCTeivEJ4B5VPIFe3jMghtqZBiEK4kBVsw3irBmjOdGWzWX8Ms73YCZ3viRmRwebGna6euUQ56GcH9OCkj2Sd9IlGh8rAHg+bmY6pQs//0/SktjWqtmXeEMLN/gdWDqzbvoQcDZH8E0pc2N93LM3U6sOE+EOKXl8QeqmVag36q4UA==; Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1sr8lt-0002kA-K0 for bug-gnu-emacs@gnu.org; Thu, 19 Sep 2024 00:28:01 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Sean Allred Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Thu, 19 Sep 2024 04:28:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 73320 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: patch Original-Received: via spool by 73320-submit@debbugs.gnu.org id=B73320.172672004810501 (code B ref 73320); Thu, 19 Sep 2024 04:28:01 +0000 Original-Received: (at 73320) by debbugs.gnu.org; 19 Sep 2024 04:27:28 +0000 Original-Received: from localhost ([127.0.0.1]:59579 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sr8lM-0002jJ-0e for submit@debbugs.gnu.org; Thu, 19 Sep 2024 00:27:28 -0400 Original-Received: from mail-il1-f182.google.com ([209.85.166.182]:60697) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1sr8lH-0002iv-Bn for 73320@debbugs.gnu.org; Thu, 19 Sep 2024 00:27:26 -0400 Original-Received: by mail-il1-f182.google.com with SMTP id e9e14a558f8ab-3a09dd37ceaso1706405ab.1 for <73320@debbugs.gnu.org>; Wed, 18 Sep 2024 21:27:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1726719960; x=1727324760; darn=debbugs.gnu.org; h=mime-version:message-id:date:references:in-reply-to:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=Jc6wqY66udr+WSJ8DzWKxfIKPwdF0DpIzspM25WBpzI=; b=gZ9vOOxbIbGNnc5WMHPOVG/k+WtlbxJd4rAhCAKDausU9AfTRcq/5kvsz89Pa95fgg EbuAVNj4mc/Rz20CphF+hO2dilg2Tfg+TjlWESgelUY3iFFMEhwH2eq7kOXtgZfQCkV7 2mupFjSaEsre8WsLXI3gVyZzzq2j5dJZ8pWduqVXiUMXhoUpHFBW3FXAAzX9urJiesMB GpWXWxLQsfN6z2xNkAQgKagMOc25wTjP0p+tEZfFUIOn8KSbz/o1uH5U7D+O3yTlUV/8 wJ0M1PEA2C5khwJ+iO7qmENVphoY+ffECjQTLthfnWtxVhi474po9eR5NtQVc3rqOIkz pbwg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1726719960; x=1727324760; h=mime-version:message-id:date:references:in-reply-to:subject:cc:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Jc6wqY66udr+WSJ8DzWKxfIKPwdF0DpIzspM25WBpzI=; b=ScIZWVX6uZd7PJzb1LWM9+kQMQY8IWaoXg9QYlzsQbX9i+RNDC82tF1jAfBiRc0yCK bP3kdlnW5cUHoW+6N7NEGteblt0FC00wmO4OpRuJWZsQkQIvxtQ+XSUKDGfbaUkt0ve0 xaACh2NQaZ87Id7PmWvK3vwqUUDDXRF8rQ5HBtzUTxkfLXXjbnCP1mBi3+xUuVXiVNi3 ffPHWirDj1Uk53FlM/Up39gpsk0WqSY0r13upRYgVOGUowREZvgqzMrjeHy7hi05SOBV 8jay7ZK19/TeeKaKb7LqApw5ezZYmLBSZ+yQx8HTLAZ8yQerQ3FnO0SA5MW3exDHFpLh 8Hlg== X-Gm-Message-State: AOJu0YzJryggRnkw0jfO2smOusbNTBTASJe/sSlttT2Ea4Yb9yum1Khn MgbgVGXJaSkBFg0mHv1xj0LiC8FlPXXDUFEPdWEG1Aeua+tZPjZoqlyqOw== X-Google-Smtp-Source: AGHT+IHHyWcS6h56WxlPdfe6NuHcCG8g0sgrFvDzujI+seYyPMKtkIsGOoxbIot/kvorvSppjiv5Qg== X-Received: by 2002:a05:6e02:1a68:b0:39f:5557:857f with SMTP id e9e14a558f8ab-3a084900bc8mr217782625ab.6.1726719960211; Wed, 18 Sep 2024 21:26:00 -0700 (PDT) Original-Received: from epic96565.epic.com (syn-075-135-086-025.res.spectrum.com. [75.135.86.25]) by smtp.gmail.com with ESMTPSA id e9e14a558f8ab-3a092e8fecasm32095455ab.80.2024.09.18.21.25.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Sep 2024 21:25:59 -0700 (PDT) In-Reply-To: <73758f39-1e18-471a-9dfb-0ceade12dacf@gutov.dev> (Dmitry Gutov's message of "Thu, 19 Sep 2024 01:27:03 +0300") X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.bugs:292024 Archived-At: Dmitry Gutov writes: >>> Yeah, I expect project-find-regexp, project-search, >>> project-query-replace-regexp might start misbehaving without >>> additional filtering -- either throwing up errors or, best case, >>> continuing to search through the "hidden" directories. >> Not sure how best to track that we should come back to this, but >> yeah. >> It seems like the right place to add some sort of switch would be in the >> `project-files` defmethod. From here, it looks like all the functions >> you mention could choose the behavior right for them. (Based on the >> function names alone -- it seems they would /also/ be interested in >> operating on only those files which exist on disk.) > > I think we can just remove the names ending with '/'. The built-in > commands don't seem to error out on them right now - probably because > there is some protection against nonexistent files - but those files > are (were) still shown as completions for project-find-file. Try out > this addition please. The performance here seems about the same even > with a large list (something I was worried about): > > diff --git a/lisp/progmodes/project.el b/lisp/progmodes/project.el > index b29d5ed5404..a2e3f3f52e6 100644 > --- a/lisp/progmodes/project.el > +++ b/lisp/progmodes/project.el > @@ -663,7 +663,7 @@ project--vc-list-files > (pcase backend > (`Git > (let* ((default-directory (expand-file-name > (file-name-as-directory dir))) > - (args '("-z")) > + (args '("-z" "--sparse")) > (vc-git-use-literal-pathspecs nil) > (include-untracked (project--value-in-dir > 'project-vc-include-untracked > @@ -703,7 +703,8 @@ project--vc-list-files > (delq nil > (mapcar > (lambda (file) > - (unless (member file submodules) > + (unless (or (member file submodules) > + (eq ?/ (aref file (1- (length file))))) > (if project-files-relative-names > file > (concat default-directory file)))) Works fine for me :-) Though I've added an additional version check inlined below. >> Incidentally looking at the version check within `project-files`, it's >> worthwhile to point out that `--sparse` is likely /not/ compatible with >> ancient versions of Git. [...] > > [...] > > We can call vc-git--program-version the same way it's used in > vc-git-state. Which version should we make the minimum? The `--sparse` option was introduced in 2.35. The following seems to work well for me: diff --git a/lisp/progmodes/project.el b/lisp/progmodes/project.el index b29d5ed5404..873bc92729d 100644 --- a/lisp/progmodes/project.el +++ b/lisp/progmodes/project.el @@ -663,7 +663,8 @@ project--vc-list-files (pcase backend (`Git (let* ((default-directory (expand-file-name (file-name-as-directory dir))) - (args '("-z")) + (args `("-z" ,@(when (version<= "2.35" (vc-git--program-version)) + '("--sparse")))) (vc-git-use-literal-pathspecs nil) (include-untracked (project--value-in-dir 'project-vc-include-untracked @@ -703,7 +704,8 @@ project--vc-list-files (delq nil (mapcar (lambda (file) - (unless (member file submodules) + (unless (or (member file submodules) + (eq ?/ (aref file (1- (length file))))) (if project-files-relative-names file (concat default-directory file)))) Since we're getting a bit busy with our conditions, though, it might be better to start using `cond`: diff --git a/lisp/progmodes/project.el b/lisp/progmodes/project.el index 873bc92729d..b42415154e3 100644 --- a/lisp/progmodes/project.el +++ b/lisp/progmodes/project.el @@ -704,11 +704,11 @@ project--vc-list-files (delq nil (mapcar (lambda (file) - (unless (or (member file submodules) - (eq ?/ (aref file (1- (length file))))) - (if project-files-relative-names - file - (concat default-directory file)))) + (cond + ((member file submodules) nil) + ((eq ?/ (aref file (1- (length file)))) nil) + (project-files-relative-names file) + (t (concat default-directory file)))) (split-string (with-output-to-string (apply #'vc-git-command standard-output 0 nil "ls-files" args)) This seems to help readability -- at least to me. There's probably also a nominal performance benefit since `cond` is a special form. I've pushed this as branch `sa/sparse-index-2` to my repository. (This is in addition to the `sa/sparse-index` branch, which contains the `file-exists-p` check mentioned below plus what might be, I take it, an ultimately unneeded opt-out parameter in `project-files`.) It's worth noting that actually performing a `file-exists-p` check here would have the added benefit of handling the awkward state between Git 2.25 (where sparse-checkout was introduced) and 2.35 (where git-ls-files learned --sparse) where ls-files could still report things that _look_ like files but are not present. This would be fixed by just replacing the (eq ..) form with (not (file-exists-p file)). -- Sean Allred