bug#61038: 30.0.50; `project-query-replace-regexp' also attempts search and replace in auto-save files

bug#61038: 30.0.50; `project-query-replace-regexp' also attempts search and replace in auto-save files

[Mickey Petersen]

I think it'd be more useful if auto-save files (or even backup files)
were excluded from all the various `project-' commands that operate on
project files.




In GNU Emacs 30.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version
 3.24.20, cairo version 1.16.0) of 2023-01-17 built on mickey-work
Repository revision: bb383a54910c3094e5d228e0af62bf70e36203ca
Repository branch: master
Windowing system distributor 'The X.Org Foundation', version 11.0.12013000
System Description: Ubuntu 20.04.3 LTS

Configured using:
 'configure --with-native-compilation --with-json --with-mailutils
 --without-compress-install --with-imagemagick CC=gcc-10'

bug#61038: 30.0.50; `project-query-replace-regexp' also attempts search and replace in auto-save files

[Dmitry Gutov]

On 24/01/2023 12:37, Mickey Petersen wrote:
> I think it'd be more useful if auto-save files (or even backup files)
> were excluded from all the various `project-' commands that operate on
> project files.

I think that depends on the project backend.

The default implementation uses grep-find-ignored-files (which include 
auto-save files, I think?).

But the vc-aware backend ignores generally what the repository 
configuration tells it to ignore. Which would generally include 
auto-save files as well, but that depends on factors external to Emacs.

What kind of backend were you trying this command with?

bug#61038: 30.0.50; `project-query-replace-regexp' also attempts search and replace in auto-save files

[Mickey Petersen]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 24/01/2023 12:37, Mickey Petersen wrote:
>> I think it'd be more useful if auto-save files (or even backup files)
>> were excluded from all the various `project-' commands that operate on
>> project files.
>
> I think that depends on the project backend.
>
> The default implementation uses grep-find-ignored-files (which include
> auto-save files, I think?).
>
> But the vc-aware backend ignores generally what the repository
> configuration tells it to ignore. Which would generally include
> auto-save files as well, but that depends on factors external to
> Emacs.
>
> What kind of backend were you trying this command with?

Default git backend. However, I'd charge that Emacs should know about
its own backup files (and not necessarily those of other Emacs
instances that may be running.)

bug#61038: 30.0.50; `project-query-replace-regexp' also attempts search and replace in auto-save files

[Dmitry Gutov]

On 25/01/2023 09:30, Mickey Petersen wrote:
> Dmitry Gutov<dgutov@yandex.ru>  writes:
> 
>> On 24/01/2023 12:37, Mickey Petersen wrote:
>>> I think it'd be more useful if auto-save files (or even backup files)
>>> were excluded from all the various `project-' commands that operate on
>>> project files.
>> I think that depends on the project backend.
>>
>> The default implementation uses grep-find-ignored-files (which include
>> auto-save files, I think?).
>>
>> But the vc-aware backend ignores generally what the repository
>> configuration tells it to ignore. Which would generally include
>> auto-save files as well, but that depends on factors external to
>> Emacs.
>>
>> What kind of backend were you trying this command with?
> Default git backend. However, I'd charge that Emacs should know about
> its own backup files (and not necessarily those of other Emacs
> instances that may be running.)

It's hard for me to say what is TRT here. If we explicitly ignore 
backups, does that mean we should add compiled objects files, etc, that 
is, everything from grep-find-ignored-files?

But I have configured my backups to be saved inside ~/.emacs.d/ 
directory, like a lot of people recommend. Via backup-directory-alist.

If you don't do that, and you have them saved side-by-side inside Git 
repository, don't you usually add .gitignore rules for them then?

So that they don't get in the way of other operations, outside of Emacs.

bug#61038: 30.0.50; `project-query-replace-regexp' also attempts search and replace in auto-save files

[Mickey Petersen]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 25/01/2023 09:30, Mickey Petersen wrote:
>> Dmitry Gutov<dgutov@yandex.ru>  writes:
>>
>>> On 24/01/2023 12:37, Mickey Petersen wrote:
>>>> I think it'd be more useful if auto-save files (or even backup files)
>>>> were excluded from all the various `project-' commands that operate on
>>>> project files.
>>> I think that depends on the project backend.
>>>
>>> The default implementation uses grep-find-ignored-files (which include
>>> auto-save files, I think?).
>>>
>>> But the vc-aware backend ignores generally what the repository
>>> configuration tells it to ignore. Which would generally include
>>> auto-save files as well, but that depends on factors external to
>>> Emacs.
>>>
>>> What kind of backend were you trying this command with?
>> Default git backend. However, I'd charge that Emacs should know about
>> its own backup files (and not necessarily those of other Emacs
>> instances that may be running.)
>
> It's hard for me to say what is TRT here. If we explicitly ignore
> backups, does that mean we should add compiled objects files, etc,
> that is, everything from grep-find-ignored-files?
>
> But I have configured my backups to be saved inside ~/.emacs.d/
> directory, like a lot of people recommend. Via backup-directory-alist.
>
> If you don't do that, and you have them saved side-by-side inside Git
> repository, don't you usually add .gitignore rules for them then?
>
> So that they don't get in the way of other operations, outside of Emacs.

(Actually this issue also afflicts auto-save files in my Emacs.)

And the files in question are not committed to the index, nor are they
part of the git tree. So they're just stray files that happen to be
important (backup, auto save) to Emacs.

It seems odd that you'd want to search and replace those by default,
particularly when Emacs is well aware of the fact that they are indeed
backups or auto saves of other files used by that instance of Emacs.

And yes indeed: why not make the project replace mechanism ignore dumb
things no one wants to edit.

And committing large, binary files to a tree is common in a wide range
of situations, though less so in Git, as it's terrible at it.

So, yes, `grep-find-ignored-files' (or a project.el equivalent) should
indeed exist.

bug#61038: 30.0.50; `project-query-replace-regexp' also attempts search and replace in auto-save files

[Dmitry Gutov]

On 25/01/2023 22:34, Mickey Petersen wrote:

> (Actually this issue also afflicts auto-save files in my Emacs.)
> 
> And the files in question are not committed to the index, nor are they
> part of the git tree. So they're just stray files that happen to be
> important (backup, auto save) to Emacs.
> 
> It seems odd that you'd want to search and replace those by default,
> particularly when Emacs is well aware of the fact that they are indeed
> backups or auto saves of other files used by that instance of Emacs.

I'm asking why they are not in your .gitignore already. They must get in 
the way of operations such as 'git status', or 'git add *', or 'git 
commit -a', or just in the way of shell completion for 'git add ...'.

> And yes indeed: why not make the project replace mechanism ignore dumb
> things no one wants to edit.

The "project replace mechanism" uses the same set of files that you get 
in completion for project-find-file. Or search through with 
'project-find-regex'.

So far the semantics of the vc-aware backend has been that all files 
that Git doesn't consider ignored (tracked or untracked) are considered 
to be part of the project.

> And committing large, binary files to a tree is common in a wide range
> of situations, though less so in Git, as it's terrible at it.

That's why people usually put the binary files, backup files, etc, in 
.gitignore.

> So, yes, `grep-find-ignored-files' (or a project.el equivalent) should
> indeed exist.

grep-find-ignored-files is a real user option already. You can also use 
project-vc-ignores, but it's nil by default.

A couple of reasons not to use grep-find-ignored-files patterns by default:

- Some users might be actually looking for one of those files, and would 
get surprised that while the Git repository lists them fine (perhaps 
they even checked in such file; maybe they're using unusual file naming 
schemes), but our project backend does not.

- Every addition to the ignored patterns is a minor but steady 
performance hit. grep-find-ignored-files has 61 element by default. 
Dropping all of those into project--vc-list-files can create a 
performance hit of an order of a magnitude. E.g. in my testing the time 
to list the files in gecko-dev went up from 1s to about 5s.

bug#61038: 30.0.50; `project-query-replace-regexp' also attempts search and replace in auto-save files

[Mickey Petersen]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 25/01/2023 22:34, Mickey Petersen wrote:
>
>> (Actually this issue also afflicts auto-save files in my Emacs.)
>> And the files in question are not committed to the index, nor are
>> they
>> part of the git tree. So they're just stray files that happen to be
>> important (backup, auto save) to Emacs.
>> It seems odd that you'd want to search and replace those by default,
>> particularly when Emacs is well aware of the fact that they are indeed
>> backups or auto saves of other files used by that instance of Emacs.
>
> I'm asking why they are not in your .gitignore already. They must get
> in the way of operations such as 'git status', or 'git add *', or 'git
> commit -a', or just in the way of shell completion for 'git add ...'.
>

Let's assume I'm simplifying a more complex workflow to aid with the
bug report.

>> And yes indeed: why not make the project replace mechanism ignore dumb
>> things no one wants to edit.
>
> The "project replace mechanism" uses the same set of files that you
> get in completion for project-find-file. Or search through with
> 'project-find-regex'.
>
> So far the semantics of the vc-aware backend has been that all files
> that Git doesn't consider ignored (tracked or untracked) are
> considered to be part of the project.
>
>> And committing large, binary files to a tree is common in a wide range
>> of situations, though less so in Git, as it's terrible at it.
>
> That's why people usually put the binary files, backup files, etc, in
> .gitignore.
>

There are many legitimate reasons for having binary files -- large
ones too -- in a repository. Though it's uncommon with git, as it does
a poor job handling them.

There are also legitimate reasons for not having expansive ignore
files, particularly with version control systems that lack the
granularity of Git and its ilk.

Nevertheless, knowing that untracked are also considered part of the
project, I can now set `project-vc-include-untracked' to nil to at
least resolve this. It would seem I was not the only one who chafed at
this edge case.

>> So, yes, `grep-find-ignored-files' (or a project.el equivalent) should
>> indeed exist.
>
> grep-find-ignored-files is a real user option already. You can also
> use project-vc-ignores, but it's nil by default.
>
> A couple of reasons not to use grep-find-ignored-files patterns by default:
>
> - Some users might be actually looking for one of those files, and
>   would get surprised that while the Git repository lists them fine
>   (perhaps they even checked in such file; maybe they're using unusual
>   file naming schemes), but our project backend does not.
>
> - Every addition to the ignored patterns is a minor but steady
>   performance hit. grep-find-ignored-files has 61 element by
>   default. Dropping all of those into project--vc-list-files can
>   create a performance hit of an order of a magnitude. E.g. in my
>   testing the time to list the files in gecko-dev went up from 1s to
>  about 5s.

Sure. But `git-grep(1)' will ignore binary files by default, for example.

bug#61038: 30.0.50; `project-query-replace-regexp' also attempts search and replace in auto-save files

[Dmitry Gutov]

On 26/01/2023 11:13, Mickey Petersen wrote:
> 
> Dmitry Gutov <dgutov@yandex.ru> writes:
> 
>> On 25/01/2023 22:34, Mickey Petersen wrote:
>>
>>> (Actually this issue also afflicts auto-save files in my Emacs.)
>>> And the files in question are not committed to the index, nor are
>>> they
>>> part of the git tree. So they're just stray files that happen to be
>>> important (backup, auto save) to Emacs.
>>> It seems odd that you'd want to search and replace those by default,
>>> particularly when Emacs is well aware of the fact that they are indeed
>>> backups or auto saves of other files used by that instance of Emacs.
>>
>> I'm asking why they are not in your .gitignore already. They must get
>> in the way of operations such as 'git status', or 'git add *', or 'git
>> commit -a', or just in the way of shell completion for 'git add ...'.
>>
> 
> Let's assume I'm simplifying a more complex workflow to aid with the
> bug report.

Okay.

> There are many legitimate reasons for having binary files -- large
> ones too -- in a repository. Though it's uncommon with git, as it does
> a poor job handling them.
> 
> There are also legitimate reasons for not having expansive ignore
> files, particularly with version control systems that lack the
> granularity of Git and its ilk.
> 
> Nevertheless, knowing that untracked are also considered part of the
> project, I can now set `project-vc-include-untracked' to nil to at
> least resolve this. It would seem I was not the only one who chafed at
> this edge case.

You can also customize project-vc-ignores to fine-tune which additional 
file to skip specifically (whether tracked or not).

But if "skip untracked" suits your mental model well, even better (it 
also increases file listing's performance).

As a default behavior, though, I think it's problematic because one 
might work for a significant amount of time on a bunch of new files 
before committing them. Depends on a workflow.

>>> So, yes, `grep-find-ignored-files' (or a project.el equivalent) should
>>> indeed exist.
>>
>> grep-find-ignored-files is a real user option already. You can also
>> use project-vc-ignores, but it's nil by default.
>>
>> A couple of reasons not to use grep-find-ignored-files patterns by default:
>>
>> - Some users might be actually looking for one of those files, and
>>    would get surprised that while the Git repository lists them fine
>>    (perhaps they even checked in such file; maybe they're using unusual
>>    file naming schemes), but our project backend does not.
>>
>> - Every addition to the ignored patterns is a minor but steady
>>    performance hit. grep-find-ignored-files has 61 element by
>>    default. Dropping all of those into project--vc-list-files can
>>    create a performance hit of an order of a magnitude. E.g. in my
>>    testing the time to list the files in gecko-dev went up from 1s to
>>   about 5s.
> 
> Sure. But `git-grep(1)' will ignore binary files by default, for example.

Hmm, I think in our case the step which will ignore the binary files is 
the search program. So (project-files) will include them in the listing, 
but both Grep and Ripgrep will skip them during the search.