bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Dmitry Gutov <dgutov@yandex.ru>]

On 08.10.2021 19:24, Nikolay Kudryavtsev wrote:
>> What's your stance of having "filemarker" projects honor (or not 
>> honor) .gitignore instructions from the containing VCS repo? 
> That's a good question. I think at any non-VC project it is fine that it 
> does not honor it, since you get what you've signed up for. I'll come 
> back to this later in this letter, since another part of this 
> conversation sort of gives the answer to it.

I didn't find a definite answer in the rest of the email, so I'll 
continue this thread here.

This is the problem: semantically, it makes sense for filemarker 
backends not to honor VC ignores. But in practice, people will often 
(probably most often) want it to, or implicitly expect it to.

Because when you are working on a subproject in a big monorepo, you 
might want to focus on its directory tree, sure, but you also expect 
that the global .gitignore settings are honored (where build artefacts, 
temporary files, etc, are configured to be skipped). You might also have 
additional per-directory .gitignore files which, again, are even more 
pertinent to the subproject, but semantically would not be expected to 
be honored.

It's not good to have conflicts like that, between semantics and 
practical requirements.

>> The problem is not being able to invent a variable, but how to honor 
>> it when listing project files. If the same backend doesn't have the 
>> information about the contained projects. 
> Yes, but at this point it's a separate feature request for a problem 
> that already exists at this point, it's just that we rely on other tools 
> to solve it for us. Let's say I maintain an application, except for a 
> module maintained by my colleague Bob. And since that module may be be 
> shared by multiple applications, it is version controlled separately. I 
> also may or may not want it to be considered a part of my project for 
> the sake of project utility commands. Here we rely on .gitignore and the 
> like. Lets say with GTAGS I can decide whether to include the bob module 
> too. But all of this works only because those backended tools happen to 
> implement this ignore functionality.

So you want to add a new backend based on GTAGS which also lists files 
by calling 'global -P' or something. It's not enough to configure the 
root-finding logic for it, you also need to implement 'project-files' 
with the above process call.

I also wouldn't want to enable such backend by default because it will 
only list files from a manually generated index, but not, say, a file 
that the user just created and saved. Not very user-friendly.

But enable or not, that's beside the point: such backend fits the 
current approach well; whoever writes it can distribute it, and the 
users who want to prioritize GTAGS over .git can put it in front.

It doesn't seem pertinent to what project-fallback should do.

> Lets say your VC is broken and just 
> won't ignore some files(happend before in practice with VC) even though 
> it ordered to? So any such project-level ignore functionality, while 
> useful and would be even more useful with more weaker custom backends, 
> is not that important in the context of this discussion in my opinion.

It doesn't really match my experience. Some VCS can be "broken" -- then 
we can ignore their equivalent of 'git ls-files' and use 'find'. We only 
have optimized paths for Git and Hg anyway. But even when we use 'find', 
we try to pick up the ignores configuration from the repo.

>> This is an inevitably slow approach. Can be bearable in a local 
>> filesystem; might be fatal over Tramp. 
> Yes. That's sort of the unfortunate price we have to pay here. I think 
> some caching can be implemented, to limit directory reading operations 
> to 1 per level.

"Some caching" has been lying around in TODO for years.

>> I'm not seeing the concrete usage scenarios yet. 
> Well think more about either the 3 marker(vc, build-tool, GTAGS) example 
> or the Bob module example. Lets try to put it in your terms of 
> "correctness". Is me wanting the build tool to define the project 
> boundary over VC "correct"? Is me possibly wanting GTAGS backend to 
> define the project boundary "correct"? Is me wanting to possibly NOT 
> exclude the Bob module from the project "correct"? Lets say I've found a 
> broken function in the general application and I now want to see whether 
> the Bob module uses it too along with any other code by calling 
> project-find-regexp. You may answer no to all of those and say that Holy 
> Correctness can be only provided by the Holy VC backend, but I don't 
> think it's a reasonable approach.

I want all of those capabilities to be reachable, but we also need to 
decide which project backends configuration is on by default.

>> These settings, as I see it, will be on the project-vc backend.
> Are you saying that if I want to declare that my make projects can be 
> subordinate to another build tool, I have to declare that in the 
> project-vc-backend of all places?

What is a 'make project'? If you have 'Makefile' inside a subdirectory 
of a GGTAGS project, do you expect the said 'make project' only to 
include the files that GGTAGS has indexed?

If yes, you need to have to integration with GGTAGS, one way or another. 
Maybe simply go through the GGTAGS backend to configure said 'make 
project', since that is the backend which is expected to honor the 
GGTAGS index.

If no, sure, have a 'make project' entry in front of 'ggtags project' 
entry in project-find-functions. That fits the current approach well.

> Again, not gonna repeat, myself on the ignore issue, apart from "you get 
> what you've signed up for".

And going back to the ignores issue: I think we shouldn't expect the 
users to know the intricacies and the semantics of the project backend 
configuration before they can use the corresponding features 
productively. The default config should fit the majority use cases in a 
predictable way.

OTOH, when you are well-acquainted with the nuts and bolts, you can 
built your own backend configuration, and you wouldn't care about the 
default setup. For such a user the current project-fallback definition 
is unnecessary: it is pretty trivial after all.

>> I think it's apparent at this point that we are venturing into the 
>> territory of pretty invasive, backward-incompatible changes to the 
>> existing project.el API. 
> True, but currently the project.el is underutilized due to its 
> simplicity and due to this we can still do reasonable invasive changes 
> to the API.

Not sure if that's true. According to the feedback, it covers the use 
cases of like 80% of the users. And the patches in this discussions 
should cover the remaining big holes which have been reported a few times.

So... I'm fine discussing additions and even big remodels, but it would 
be better to consider those piece-by-piece, and probably in separate bug 
reports. We should still keep an eye out for performance, though, and 
the OOtB experience.

>> Right, yeah. Having module detection on a hook seems more flexible 
>> than putting it on a method dispatches by project type, because then 
>> the Maven extension can work with any project backend, not just some 
>> specific Maven-supporting one. 
> The possible actions for Maven would already work any project backend, 
> it's the question of how to ensure the proper root for it.

Yes.

>> FWIW, Steinar asked for being able to choose whether to act on a 
>> module or on a project, and for that such separation seems to be 
>> necessary. 
> 
> I don't think he cares about Emacs having a distinction, more about the 
> abiilty to independently act on(compile) those parts.

If the user wants to choose what to act on (whole project or current 
module), the information about modules needs to be discovered as a 
separate semantic unit. Not sure how else it would be possible to 
implement such choice in user interaction.

>> Debugging seems to be a separate feature from build tools (requiring 
>> its own information, and a fair amount of it). Although, depending on 
>> a language and environment, it might require integration with build 
>> tools as well.
> Yeah, and that's one of the conceptual problems with that approach: 
> assuming too much. Project markers(or "build-tools" markers in your 
> paradigm) are not limited to build tools, it may be a tags tool for 
> example.

Only if you intend to use the 'tags tool' as a build tool, e.g. list the 
available tasks or invoke one (for example, to rebuild the index). Then 
said tags tool can be put into build-tools-functions, with appropriate 
implementations for 'list tasks' and 'run task'.

> Even take Zhou's patch. Lets operate on the assumption that we cannot 
> let anyone go astray from the paradise of the Holy VCs correctness. 
> Every single problem you've mentioned would befall on those trying to 
> use a plain project. Adding a new backend would just let the heathens 
> win, since they can add those files anywhere and stray away from the 
> light. From such point of view, you can never add new backends, you can 
> only add new build tools.

You can add new backends: you can add a bridge to Projectile, for 
example (there is one inside one of the reports in its issue tracker). 
Or you can create the backend based on GGTAGS, the one you described.

As long as it's feature-rich enough, and the behaviors are documented, 
so the users know what to expect from it, it's totally fine.

But both of those are more complex than the project-plain backend Zhou 
originally proposed, and the project-fallback in the latest patch.

> Or to put it another way, whenever people want to mess with project 
> backends, what they want is to define a scope in which they want to 
> operate. That scope may differ from the scope defined by VC.

I'd like to point out that this is already easy to do.

> Also 
> there's nothing wrong in VC scope being available at the same time at 
> the user declared project scope. What I mean by that is that nobody 
> would complain if every project-* utility commands gets a sister named 
> project-vc-* that would allow you to the same operation, but explicitly 
> within the VC scope of the current project.

Do we really want to ask everyone to use a separate set of commands to 
apply the 'ignores' config from the current VCS repo?

I mean, it's nice to have the option for certain cases, but one would 
imagine this behavior would be more desirable by default, no?

>> The 'project-vc-subprojects' patch.
> This a solution to the subprojects hiding problem discussed earlier, 
> perfectly acceptable for me, but as I mentioned before is not something 
> I'd use much personally.

FWIW, a user option 'do not exclude subprojects' could very easily be 
added as well.