bug#41572: 28.0.50; [PATCH] Support plain project marked with file .emacs-project

[Dmitry Gutov <dgutov@yandex.ru>]

On 07.10.2021 16:08, Nikolay Kudryavtsev wrote:
>> When should we not do it? 
> Personal preference. I'd probably never hide subproject files in 99% of 
> the projects I work on.

All right. So you might like the possible alternative approach to this 
problem.

What's your stance of having "filemarker" projects honor (or not honor) 
.gitignore instructions from the containing VCS repo?

>> But then you will end up specifying the same information twice. Once 
>> when setting up those new backends -- and the second time when 
>> configuring the parent project to ignore particular subdirectories.
> You can have a `project-hide-nested-project-files' variable, set it once 
> for all backends. Maybe per backend version too.

The problem is not being able to invent a variable, but how to honor it 
when listing project files. If the same backend doesn't have the 
information about the contained projects.

>> why not have a single backend for that purpose, with a custom var 
>> containing the list of files names?
> Right, so remember, I said we can use this to realize most of the 
> Projectile backends? Well, the ones we can't realize would require 
> regexps, usually of the "\\.ext$" kind. So you have to account for those 
> too. Then there may be some other possible cases requiring custom function.

project-fallback-markers can easily support globs (I didn't add the 
support initially because the naive implementation of checking for files 
including globs is slower). Support for predicate functions among its 
entries is pretty trivial to do.

OTOH, when we only have a list of project-finding functions (one per 
project marker, say), it's pretty difficult to find the deepest 
enclosing project directory without running them all, and having them 
traverse the parent directories up until /. This is an inevitably slow 
approach. Can be bearable in a local filesystem; might be fatal over Tramp.

> Lets say I have this structure: VC root, subfolders containing multiple 
> related, but independent projects in some programming language, backend 
> for which exists; deeper in one of the subfolders I have a GTAGS file, 
> assume GTAGS backend exists in one form or another. GTAGS file is placed 
> in this location because elsewhere in the repo there are symbol names 
> that are too close to each other and it's more convenient not having 
> them show up when I lookup something. I don't want VC backend to define 
> the project root here for obvious reasons. The major mode build tool 
> backend should do OK and I may or may not want GTAGS to define the 
> project root.

It doesn't seem like you want GTAGS to define project root; you'll only 
want certain commands, like xref-find-definitions, to use the closes 
GTAGS file.

Good thing xref-find-definitions doesn't use the project.el 
infrastructure at all.

> Having one find-function list which the user can reorder as he sees fit 
> and that list may contain not just filenames, but regexps and custom 
> functions too.

I'm not seeing the concrete usage scenarios yet.

> As for customizability: we're already discussing at least 2 backend 
> settings: hide-nested-project-files and recursivity. Those settings 
> require a backend to be something more than a filename string in a 
> secondary list.

These settings, as I see it, will be on the project-vc backend.

>> That didn't really answer my question. 
> All right, lets rephrase the answer. At the moment in time a backend is 
> defined we do not know every single exact situation that backend would 
> have to operate in, because that would require the ability to predict 
> time, which we technologically do not have at the current moment. Lets 
> say I add a backend for my major mode. Someone in exactly 18 days, 6 
> hours, 5 minutes and 3 seconds decides to use it to work on his project. 
> Unfortunately I do not know whether his project is in VCd and if VC 
> project backend returns the same root. If I could predict time and my 
> prediction of all possible future use cases would show that VC backend 
> returns the same root for every single one of them, I of course would 
> not bother adding mine. But because my imperfect human understanding 
> makes me think that it won't, I have to add a backend of my own.

Okay, but we need both correctness and speed. Answering "see if there is 
a fast backend behind this one with the same root" is not very useful, 
since in this scenario we seemingly (!) don't need the first backend anyway.

But perhaps you meant we should always scan the full list of backends, 
and we should not only accept a "fast" backend with the same root, but 
with root in any parent directory as well. Which would correspond to the 
popular situation with the monorepo.

Setting aside the performance concerns of always scanning for all 
backends in project-find-functions, what do we do with the ignore 
entries? A backend is (very roughly) defined as (root . ignores), to be 
able to only list and work with a subset of files from its root's 
directory tree.

The VC backend traditionally includes .gitignore entries in its list of 
ignores. So its project-ignores method includes them, and its 
project-files method implicitly uses them.

But what if your first detected backend has a different list of ignores 
configured? Or none? Do we account for the latter's (fast) backend 
ignores when asking it for the list of files in the first backend's 
detected root directory. In practical terms, as just one example, that 
would mean honoring .gitignore at the top of a monorepo. Which might 
seem like a good thing to do, but also a somewhat unexpected behavior, 
conceptually.

> Probably the best route here globally, is changing 
> project-find-functions to a list with numerical priorities, so that you 
> could set VC backend to priority 0 in your init and instruct mode 
> developers to never put anything with the same priority or higher in the 
> docstring.

That's quite easy to do already: project-find-functions is a hook, and 
'add-hook' has a DEPTH argument. Since Emacs 27, I think.

>> That's possible, but it's not at all a guarantee that in every big 
>> project every Makefile will have a "dominating" Makefile of its own. 
> Yes, but we can define a list of possible parent backends for every 
> backend. For example you could set VC as possible parent backend for 
> Makefile. Would probably not be a good idea in general, but for your 
> VC-first workflow, should be fine.

I think it's apparent at this point that we are venturing into the 
territory of pretty invasive, backward-incompatible changes to the 
existing project.el API.

>> It would seem like your vision of the project could benefit from a 
>> notion like "facet". E.g. a project lookup would search for not just 
>> "the current project", but "the current build project" or "current 
>> file listing project", or... I don't know what else, actually. But I'm 
>> sure there can be other additions (something test-framework related, 
>> maybe). 
> Or a "module" right? I was thinking about this too, and could not find a 
> good name for it either.

Right, yeah. Having module detection on a hook seems more flexible than 
putting it on a method dispatches by project type, because then the 
Maven extension can work with any project backend, not just some 
specific Maven-supporting one.

> Such a solution would be a reliable working 
> compromise between our schools of thought. You get your project-root 
> untouched, I get my own project root to do whatever I please with it. 
> The problem with it is that it's really overengineered. For most 
> projects there would be a 1 to 1 relationship between the project and 
> the module(artifact?) and even if it's not 1 to 1, there's a root module 
> most of the time. That's why it feels inferior to me in comparison to 
> just treating everything as projects and going bottom up.

I don't know if it's really overengineered, especially compared to 
certain proposals in your previous email.

FWIW, Steinar asked for being able to choose whether to act on a module 
or on a project, and for that such separation seems to be necessary.

Thanks for the reminder about that thread, BTW.

Speaking of build tools -- just as well, if the tool is in the same 
directory as the project root, you don't need any additional backends. 
But if you do need extra detection, file-finding logic, that could be 
put into buildtool-find-functions.

>> Which would return, for example, new kind of object, and that object 
>> could tell the parent directory of its build file, and the list of the 
>> tasks described in it, and... perhaps something about how those tasks 
>> should be invoked. That new abstraction could be used by commands that 
>> want to interact with build files in an abstract fashion and to launch 
>> build tasks. 
> After studying Projectile build commands I found them inadequate, since 
> it provides just two, compile and build, while even my simple major mode 
> requires a separate debugging command too.

Debugging seems to be a separate feature from build tools (requiring its 
own information, and a fair amount of it). Although, depending on a 
language and environment, it might require integration with build tools 
as well.

> This solution suffers from 
> the same lack of flexibility. Take for example unit testing. It's not 
> necessarily build tool subservient and can be independent from it, but 
> it is situated in relation to the build tool root.

So what's the problem with this approach, then? The run-tests command 
can for example run the buildtool-find-functions, take the appropriate 
one, search its parent directory for the presence of testing frameworks, 
and launch it.

> The maven hierarchy 
> is problematic for this too, while my "treat everything as projects" 
> paradigm has an elegant solution in which you can use a numerical prefix 
> to launch a build command on the Nth parent of the current project.

It sounds like this approach, at the very least, doesn't go through 
'project-current'?

Are there other advantages, rather than being able to choose the project 
parent nesting depth with the prefix argument?

>> There are two patches in this bug report. Have you looked at the other 
>> one?
> You mean the original patch? Well it is IMHO better than yours since 
> it's less ambitious and does not go in what I believe to be the wrong 
> direction.

The 'project-vc-subprojects' patch. Here:

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=41572#85