Eglot, project.el, and python virtual environments

Eglot, project.el, and python virtual environments

[Eric Abrahamsen]

Hi,

Here's another issue that's technically a emacs.help question, but might
result in some code/documentation updates, so I'm sending it here.

My main day-job code base is an AWS CloudFormation monstrosity involving
several Python Lambdas, among other things. Basic project structure
looks like:

project_root
├── .git
├── src
│   └── python
│       ├── VeryImportantLambda
│       │   └── .venv
│       ├── MoreImportance
│       │   └── .venv
│       ├── RunInCaseOfEmergency
│       │   └── .venv

I'm using the python-lsp-server python package in each Python
subdirectory, and the key is that each of those directories is a virtual
environment that needs to stay isolated from the others. Each has
different packages installed, and in some cases even the Python versions
are different (though I'm trying to get rid of that).

When I was using lsp-mode this wasn't difficult, because lsp-mode and
project.el are essentially orthogonal: if I visited a python file in a
given lambda directory, I could use `pyvenv-activate' to activate that
environment, and then the `lsp' invocation would confine itself to
python files within the environment. Project.el just provided
project-wide navigation.

Now I'm trying to move to Eglot, and there is tighter integration
between Eglot and project.el. Turning on Eglot in one lambda starts the
server for all Python libraries in the whole project, not just the
current environment. I looked into constructing my own version of the
call to `eglot', but it is tightly tied to a project, all the way down.

Is anyone else handling this situation? Any suggestions how to make it
work?

Thanks,
Eric

Re: Eglot, project.el, and python virtual environments

[Danny Freeman]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Hi,
>
> Here's another issue that's technically a emacs.help question, but might
> result in some code/documentation updates, so I'm sending it here.
>
> My main day-job code base is an AWS CloudFormation monstrosity involving
> several Python Lambdas, among other things. Basic project structure
> looks like:
>
> project_root
> ├── .git
> ├── src
> │   └── python
> │       ├── VeryImportantLambda
> │       │   └── .venv
> │       ├── MoreImportance
> │       │   └── .venv
> │       ├── RunInCaseOfEmergency
> │       │   └── .venv
>
> I'm using the python-lsp-server python package in each Python
> subdirectory, and the key is that each of those directories is a virtual
> environment that needs to stay isolated from the others. Each has
> different packages installed, and in some cases even the Python versions
> are different (though I'm trying to get rid of that).
>
> When I was using lsp-mode this wasn't difficult, because lsp-mode and
> project.el are essentially orthogonal: if I visited a python file in a
> given lambda directory, I could use `pyvenv-activate' to activate that
> environment, and then the `lsp' invocation would confine itself to
> python files within the environment. Project.el just provided
> project-wide navigation.
>
> Now I'm trying to move to Eglot, and there is tighter integration
> between Eglot and project.el. Turning on Eglot in one lambda starts the
> server for all Python libraries in the whole project, not just the
> current environment. I looked into constructing my own version of the
> call to `eglot', but it is tightly tied to a project, all the way down.
>
> Is anyone else handling this situation? Any suggestions how to make it
> work?
>
> Thanks,
> Eric


I have NOT been in this situation, but it sounds like you want to keep
using project.el as is, but override eglot's usage of project.el to
identify root directory to start a lsp server in.

I think the place to start looking would be the function
`eglot--current-project`. 

It uses a var called `eglot-lsp-context` which is `t` when eglot is
searching for a project. 

Knowing this, there is another var called `project-find-functions` that
project.el uses when searching for project roots, which can be used for
finding custom project roots. I have one that looks for a file named
".project.el". If that file exists then that directory is identified as
a project. It's really useful when you have a project not in source
control. 

See:
```
(defun project-find-project.el (dir)
  "Returns a `manual-project' instance if the project of the current
DIR has a .project.el file in its root directory."
  (let ((root (locate-dominating-file dir ".project.el")))
    (when root
      (cons 'transient root))))

(add-hook 'project-find-functions #'project-find-project.el)
```

Now, I am not a python programmer, but lets pretend that python
virtualenviroment depends on a file located in the directory you want to
activate one of these virtual environments, say `.virtualenv`.

You can write a project-find-functions implementation that looks for
these virutalenv files ONLY when eglot-lsp-context is active.

```
(defun project-find-virtualenv-for-eglot (dir)
  (when eglot-lsp-context
    (let ((root (locate-dominating-file dir ".virtualenv")))
      (when root
        (cons 'transient root)))))

(add-hook 'project-find-functions #'project-find-virtualenv-for-eglot)
```

I did not test this, but I think it should send you down the right path.
Eglot should see your aws lambda folders as the project root, and
project.el should see the parent.

If python virtual environments do not have any kind of file marker in
the filesystem, you could use a dummy file in those directories like
`.eglot-project` or something.

Hope this helps,
-- 
Danny Freeman

Re: Eglot, project.el, and python virtual environments

[Eric Abrahamsen]

Danny Freeman <danny@dfreeman.email> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>

[...]

>> Now I'm trying to move to Eglot, and there is tighter integration
>> between Eglot and project.el. Turning on Eglot in one lambda starts the
>> server for all Python libraries in the whole project, not just the
>> current environment. I looked into constructing my own version of the
>> call to `eglot', but it is tightly tied to a project, all the way down.
>>
>> Is anyone else handling this situation? Any suggestions how to make it
>> work?
>>
>> Thanks,
>> Eric
>
>
> I have NOT been in this situation, but it sounds like you want to keep
> using project.el as is, but override eglot's usage of project.el to
> identify root directory to start a lsp server in.
>
> I think the place to start looking would be the function
> `eglot--current-project`. 
>
> It uses a var called `eglot-lsp-context` which is `t` when eglot is
> searching for a project. 
>
> Knowing this, there is another var called `project-find-functions` that
> project.el uses when searching for project roots, which can be used for
> finding custom project roots. I have one that looks for a file named
> ".project.el". If that file exists then that directory is identified as
> a project. It's really useful when you have a project not in source
> control. 
>
> See:
>
> ```
> (defun project-find-project.el (dir)
>   "Returns a `manual-project' instance if the project of the current
> DIR has a .project.el file in its root directory."
>   (let ((root (locate-dominating-file dir ".project.el")))
>     (when root
>       (cons 'transient root))))
>
> (add-hook 'project-find-functions #'project-find-project.el)
> ```
>
>
> Now, I am not a python programmer, but lets pretend that python
> virtualenviroment depends on a file located in the directory you want to
> activate one of these virtual environments, say `.virtualenv`.
>
> You can write a project-find-functions implementation that looks for
> these virutalenv files ONLY when eglot-lsp-context is active.
>
> ```
> (defun project-find-virtualenv-for-eglot (dir)
>   (when eglot-lsp-context
>     (let ((root (locate-dominating-file dir ".virtualenv")))
>       (when root
>         (cons 'transient root)))))
>
> (add-hook 'project-find-functions #'project-find-virtualenv-for-eglot)
> ```
>
> I did not test this, but I think it should send you down the right path.
> Eglot should see your aws lambda folders as the project root, and
> project.el should see the parent.
>
> If python virtual environments do not have any kind of file marker in
> the filesystem, you could use a dummy file in those directories like
> `.eglot-project` or something.
>
> Hope this helps,

Thank you very much, that does help indeed! I started off down this path
and had gotten as far as eglot-lsp-context. Then I realized I couldn't
just return a directory, it had to be a project object, and that's when
I decided someone else must have a better approach. But it's great to
see all this laid out, and knowing about `(cons 'transient root)' is
very helpful, as well.

I'll play around with this a bit. It seems like a not-outlandish
situation, and I wonder if it might not be useful to provide some
standardized method of helping Eglot find an alternate project.

Thanks again,
Eric

Re: Eglot, project.el, and python virtual environments

[Philip Kaludercic]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Hi,
>
> Here's another issue that's technically a emacs.help question, but might
> result in some code/documentation updates, so I'm sending it here.
>
> My main day-job code base is an AWS CloudFormation monstrosity involving
> several Python Lambdas, among other things. Basic project structure
> looks like:
>
> project_root
> ├── .git
> ├── src
> │   └── python
> │       ├── VeryImportantLambda
> │       │   └── .venv
> │       ├── MoreImportance
> │       │   └── .venv
> │       ├── RunInCaseOfEmergency
> │       │   └── .venv
>
> I'm using the python-lsp-server python package in each Python
> subdirectory, and the key is that each of those directories is a virtual
> environment that needs to stay isolated from the others. Each has
> different packages installed, and in some cases even the Python versions
> are different (though I'm trying to get rid of that).
>
> When I was using lsp-mode this wasn't difficult, because lsp-mode and
> project.el are essentially orthogonal: if I visited a python file in a
> given lambda directory, I could use `pyvenv-activate' to activate that
> environment, and then the `lsp' invocation would confine itself to
> python files within the environment. Project.el just provided
> project-wide navigation.
>
> Now I'm trying to move to Eglot, and there is tighter integration
> between Eglot and project.el. Turning on Eglot in one lambda starts the
> server for all Python libraries in the whole project, not just the
> current environment. I looked into constructing my own version of the
> call to `eglot', but it is tightly tied to a project, all the way down.
>
> Is anyone else handling this situation? Any suggestions how to make it
> work?

I am not certain if this is the right tool for the job, but have you
taken a look at `buffer-env'?  I have used it in combination with Eglot,
but the problem I was tackling was not the same as yours.

> Thanks,
> Eric

Re: Eglot, project.el, and python virtual environments

[Eric Abrahamsen]

Philip Kaludercic <philipk@posteo.net> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> Hi,
>>
>> Here's another issue that's technically a emacs.help question, but might
>> result in some code/documentation updates, so I'm sending it here.
>>
>> My main day-job code base is an AWS CloudFormation monstrosity involving
>> several Python Lambdas, among other things. Basic project structure
>> looks like:
>>
>> project_root
>> ├── .git
>> ├── src
>> │   └── python
>> │       ├── VeryImportantLambda
>> │       │   └── .venv
>> │       ├── MoreImportance
>> │       │   └── .venv
>> │       ├── RunInCaseOfEmergency
>> │       │   └── .venv
>>
>> I'm using the python-lsp-server python package in each Python
>> subdirectory, and the key is that each of those directories is a virtual
>> environment that needs to stay isolated from the others. Each has
>> different packages installed, and in some cases even the Python versions
>> are different (though I'm trying to get rid of that).
>>
>> When I was using lsp-mode this wasn't difficult, because lsp-mode and
>> project.el are essentially orthogonal: if I visited a python file in a
>> given lambda directory, I could use `pyvenv-activate' to activate that
>> environment, and then the `lsp' invocation would confine itself to
>> python files within the environment. Project.el just provided
>> project-wide navigation.
>>
>> Now I'm trying to move to Eglot, and there is tighter integration
>> between Eglot and project.el. Turning on Eglot in one lambda starts the
>> server for all Python libraries in the whole project, not just the
>> current environment. I looked into constructing my own version of the
>> call to `eglot', but it is tightly tied to a project, all the way down.
>>
>> Is anyone else handling this situation? Any suggestions how to make it
>> work?
>
> I am not certain if this is the right tool for the job, but have you
> taken a look at `buffer-env'?  I have used it in combination with Eglot,
> but the problem I was tackling was not the same as yours.

I hadn't seen that before, thanks. But it's pretty much what pvenv and
poetry-mode (and probably others) already do: make sure that a python
virtual environment is "activated" for buffers in a particular directory
tree.

With Danny's nudging I've nearly got this working, except for the
virtual environment aspect: once I've run Eglot in one python-mode
buffer, any time I switch to another python-mode buffer Eglot runs
immediately, before I've had time to activate the virtual environment,
and so fails to find the language server executable.

I'm sure with the proper use of hooks I'll be able to squeeze the
`pyvenv-activate' call in before Eglot runs.

Eric

Re: Eglot, project.el, and python virtual environments

[North Year]

[-- Attachment #1: Type: text/plain, Size: 2959 bytes --]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Hi,
>
> Here’s another issue that’s technically a emacs.help question, but might
> result in some code/documentation updates, so I’m sending it here.
>
> My main day-job code base is an AWS CloudFormation monstrosity involving
> several Python Lambdas, among other things. Basic project structure
> looks like:
>
> project_root
> ├── .git
> ├── src
> │   └── python
> │       ├── VeryImportantLambda
> │       │   └── .venv
> │       ├── MoreImportance
> │       │   └── .venv
> │       ├── RunInCaseOfEmergency
> │       │   └── .venv
>
> I’m using the python-lsp-server python package in each Python
> subdirectory, and the key is that each of those directories is a virtual
> environment that needs to stay isolated from the others. Each has
> different packages installed, and in some cases even the Python versions
> are different (though I’m trying to get rid of that).

Maybe I have a silly solution,

Since you are using .git, then maybe you can create a .hg, .svn,
or what ever CVS tools other than svn,
in VeryImportantLambda folder, MoreImportance folder, and RunInCaseOfEmergency folder.

Then when you visit files in VeryImportantLambda folder,
then project.el will set the project root at VeryImportantLambda,
I think everything would just be fine there.
As long as you don’t launch eglot python lsp in the “actual” project (which is .git),
which for example is project_root/src/python/main.py
then everything should be fine.

Just have a try
where test is a .git folder and test/test2 and test/test3 are two .hg folder,
then eglot will launch two lsps at test2 and test3 folder.

You can just add .hg to your .gitignore and magit can used as usual with no pain.

This is a really stupid but I think with not too much effort try.

――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
[eglot] Connected! Server `EGLOT (test3/python-mode)’ now managing `python-mode’ buffers in project `test3’.
[eglot] (warning) Server tried to register unsupported capability `workspace/didChangeWorkspaceFolders’
Wrote /Users/me/Downloads/test/test3/cat.py
“test3/cat.py” 1L, 23C written
dog.py has auto save data; consider M-x recover-this-file
[eglot] Connected! Server `EGLOT (test2/python-mode)’ now managing `python-mode’ buffers in project `test2’.
[eglot] (warning) Server tried to register unsupported capability `workspace/didChangeWorkspaceFolders’
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――

Re: Eglot, project.el, and python virtual environments

[Danny Freeman]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> > > project_root
> > > ├── .git
> > > ├── src
> > > │   └── python
> > > │       ├── VeryImportantLambda
> > > │       │   └── .venv
> > > │       ├── MoreImportance
> > > │       │   └── .venv
> > > │       ├── RunInCaseOfEmergency
> > > │       │   └── .venv

I just realized you posted information about the .venv files in your
original post and I completely overlooked that lol. 

> Thank you very much, that does help indeed! I started off down this path
> and had gotten as far as eglot-lsp-context. Then I realized I couldn't
> just return a directory, it had to be a project object, and that's when
> I decided someone else must have a better approach. But it's great to
> see all this laid out, and knowing about `(cons 'transient root)' is
> very helpful, as well.
>
> I'll play around with this a bit. It seems like a not-outlandish
> situation, and I wonder if it might not be useful to provide some
> standardized method of helping Eglot find an alternate project.
>
> Thanks again,
> Eric

Anyways yeah, I think I've heard about someone else running into this as
well. Maybe it was on the reddit emacs forum. Either way, I do not think
you are the first.

What might an extension for Eglot look like? Maybe a built in
`project-find-functions' implementation that does something similar
looking for a `.eglot-lsp-root` (or probably something configurable) as
a transient project root? I don't know if that would cause problems out
of the box for some people, but it seems safe.

-- 
Danny Freeman

Re: Eglot, project.el, and python virtual environments

[Eric Abrahamsen]

Danny Freeman <danny@dfreeman.email> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> > > project_root
>> > > ├── .git
>> > > ├── src
>> > > │   └── python
>> > > │       ├── VeryImportantLambda
>> > > │       │   └── .venv
>> > > │       ├── MoreImportance
>> > > │       │   └── .venv
>> > > │       ├── RunInCaseOfEmergency
>> > > │       │   └── .venv
>
> I just realized you posted information about the .venv files in your
> original post and I completely overlooked that lol. 
>
>> Thank you very much, that does help indeed! I started off down this path
>> and had gotten as far as eglot-lsp-context. Then I realized I couldn't
>> just return a directory, it had to be a project object, and that's when
>> I decided someone else must have a better approach. But it's great to
>> see all this laid out, and knowing about `(cons 'transient root)' is
>> very helpful, as well.
>>
>> I'll play around with this a bit. It seems like a not-outlandish
>> situation, and I wonder if it might not be useful to provide some
>> standardized method of helping Eglot find an alternate project.
>>
>> Thanks again,
>> Eric
>
> Anyways yeah, I think I've heard about someone else running into this as
> well. Maybe it was on the reddit emacs forum. Either way, I do not think
> you are the first.
>
> What might an extension for Eglot look like? Maybe a built in
> `project-find-functions' implementation that does something similar
> looking for a `.eglot-lsp-root` (or probably something configurable) as
> a transient project root? I don't know if that would cause problems out
> of the box for some people, but it seems safe.

That was my first thought: a file cookie that told Eglot to consider
this directory a project root.

But I don't know enough about Eglot and its internals to be sure. I'm
not even really sure what Eglot uses a project definition for: I guess
to decide which buffers in a given major-mode should use the same
invocation of a language server protocol? One process used in all those
project buffers? Is that even how it groups things? I don't know.

The core issue in my case is that I need to be able to tell Eglot to use
a separate language server executable for all python-mode buffers within
a certain directory tree. Maybe there is a simpler way of doing that
that doesn't require me to trick Eglot into thinking the project is
smaller than it is. All a virtual environment does, fundamentally, is
munge PATH so that Python-related executables are found locally, rather
than system-wide. Maybe there's a relatively simple way to explicitly
tell Eglot to do that.

I think the approach I'm on now can probably be made to work, but it
would be nice to know if there's a simpler way. I might spend a bit of
time looking at Eglot's source, and open a bug report.

Thanks!
Eric

Re: Eglot, project.el, and python virtual environments

[Eric Abrahamsen]

North Year <ny-ml@outlook.com> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> Hi,
>>
>> Here’s another issue that’s technically a emacs.help question, but might
>> result in some code/documentation updates, so I’m sending it here.
>>
>> My main day-job code base is an AWS CloudFormation monstrosity involving
>> several Python Lambdas, among other things. Basic project structure
>> looks like:
>>
>> project_root
>> ├── .git
>> ├── src
>> │   └── python
>> │       ├── VeryImportantLambda
>> │       │   └── .venv
>> │       ├── MoreImportance
>> │       │   └── .venv
>> │       ├── RunInCaseOfEmergency
>> │       │   └── .venv
>>
>> I’m using the python-lsp-server python package in each Python
>> subdirectory, and the key is that each of those directories is a virtual
>> environment that needs to stay isolated from the others. Each has
>> different packages installed, and in some cases even the Python versions
>> are different (though I’m trying to get rid of that).
>
> Maybe I have a silly solution,
>
> Since you are using .git, then maybe you can create a .hg, .svn,
> or what ever CVS tools other than svn,
> in VeryImportantLambda folder, MoreImportance folder, and RunInCaseOfEmergency folder.
>
> Then when you visit files in VeryImportantLambda folder,
> then project.el will set the project root at VeryImportantLambda,
> I think everything would just be fine there.
> As long as you don’t launch eglot python lsp in the “actual” project (which is .git),
> which for example is project_root/src/python/main.py
> then everything should be fine.

Ah, but I'm trying to have my cake and eat it, too: have project.el
consider the whole codebase as a single project, but have Eglot use the
language server executables from the sub-directories. Putting a dummy
CVS file in the subdirectories would fool both Eglot and project.el.

Re: Eglot, project.el, and python virtual environments

[Yuan Fu]

> On Nov 17, 2022, at 10:06 AM, Eric Abrahamsen <eric@ericabrahamsen.net> wrote:
> 
> Danny Freeman <danny@dfreeman.email> writes:
> 
>> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>> 
>>>>> project_root
>>>>> ├── .git
>>>>> ├── src
>>>>> │   └── python
>>>>> │       ├── VeryImportantLambda
>>>>> │       │   └── .venv
>>>>> │       ├── MoreImportance
>>>>> │       │   └── .venv
>>>>> │       ├── RunInCaseOfEmergency
>>>>> │       │   └── .venv
>> 
>> I just realized you posted information about the .venv files in your
>> original post and I completely overlooked that lol. 
>> 
>>> Thank you very much, that does help indeed! I started off down this path
>>> and had gotten as far as eglot-lsp-context. Then I realized I couldn't
>>> just return a directory, it had to be a project object, and that's when
>>> I decided someone else must have a better approach. But it's great to
>>> see all this laid out, and knowing about `(cons 'transient root)' is
>>> very helpful, as well.
>>> 
>>> I'll play around with this a bit. It seems like a not-outlandish
>>> situation, and I wonder if it might not be useful to provide some
>>> standardized method of helping Eglot find an alternate project.
>>> 
>>> Thanks again,
>>> Eric
>> 
>> Anyways yeah, I think I've heard about someone else running into this as
>> well. Maybe it was on the reddit emacs forum. Either way, I do not think
>> you are the first.
>> 
>> What might an extension for Eglot look like? Maybe a built in
>> `project-find-functions' implementation that does something similar
>> looking for a `.eglot-lsp-root` (or probably something configurable) as
>> a transient project root? I don't know if that would cause problems out
>> of the box for some people, but it seems safe.
> 
> That was my first thought: a file cookie that told Eglot to consider
> this directory a project root.
> 
> But I don't know enough about Eglot and its internals to be sure. I'm
> not even really sure what Eglot uses a project definition for: I guess
> to decide which buffers in a given major-mode should use the same
> invocation of a language server protocol? One process used in all those
> project buffers? Is that even how it groups things? I don't know.

AFAIK eglot uses project.el for finding projects.

> 
> The core issue in my case is that I need to be able to tell Eglot to use
> a separate language server executable for all python-mode buffers within
> a certain directory tree. Maybe there is a simpler way of doing that
> that doesn't require me to trick Eglot into thinking the project is
> smaller than it is. All a virtual environment does, fundamentally, is
> munge PATH so that Python-related executables are found locally, rather
> than system-wide. Maybe there's a relatively simple way to explicitly
> tell Eglot to do that.

You can add dir-local variables and set eglot-server-programs for each directory differently, each mapping python-mode to different server executables. (And also use .project.el cookies to limit the scope of the project.)

> 
> I think the approach I'm on now can probably be made to work, but it
> would be nice to know if there's a simpler way. I might spend a bit of
> time looking at Eglot's source, and open a bug report.
> 
> Thanks!
> Eric

Yuan

Re: Eglot, project.el, and python virtual environments

[Tim Cross]

Danny Freeman <danny@dfreeman.email> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> > > project_root
>> > > ├── .git
>> > > ├── src
>> > > │   └── python
>> > > │       ├── VeryImportantLambda
>> > > │       │   └── .venv
>> > > │       ├── MoreImportance
>> > > │       │   └── .venv
>> > > │       ├── RunInCaseOfEmergency
>> > > │       │   └── .venv
>
> I just realized you posted information about the .venv files in your
> original post and I completely overlooked that lol. 
>
>> Thank you very much, that does help indeed! I started off down this path
>> and had gotten as far as eglot-lsp-context. Then I realized I couldn't
>> just return a directory, it had to be a project object, and that's when
>> I decided someone else must have a better approach. But it's great to
>> see all this laid out, and knowing about `(cons 'transient root)' is
>> very helpful, as well.
>>
>> I'll play around with this a bit. It seems like a not-outlandish
>> situation, and I wonder if it might not be useful to provide some
>> standardized method of helping Eglot find an alternate project.
>>
>> Thanks again,
>> Eric
>
> Anyways yeah, I think I've heard about someone else running into this as
> well. Maybe it was on the reddit emacs forum. Either way, I do not think
> you are the first.
>
> What might an extension for Eglot look like? Maybe a built in
> `project-find-functions' implementation that does something similar
> looking for a `.eglot-lsp-root` (or probably something configurable) as
> a transient project root? I don't know if that would cause problems out
> of the box for some people, but it seems safe.

Just a thought, apologies if I've missed something as I'm coming to this
late....

I"m making the basic assumption that a project, from the eglot
perspective, is what project.el tells it is a project. 

Is this something which should be addressed at the eglot level or at the
project level? It seems to me that what you really have here is a main
project which consists of a number of sub-projects and that it could be
beneficial more generally if you could more easily isolate out the
sub-projects (not just for eglot, but possibly for other tools as well)
at the project.el layer rather than the eglot layer. This could mean the
user has to be more involved in defining the project and sub-project
structure/relationships, but that may be reasonable when you have more
complex project structure?

I guess my question here is whether the focus should be on enhancing
project.el rather than modifying/enhancing eglot.el to handle this use
case?

Re: Eglot, project.el, and python virtual environments

[Phil Sainty]

On 2022-11-18 11:21, Tim Cross wrote:
> Is this something which should be addressed at the eglot level or at 
> the
> project level? It seems to me that what you really have here is a main
> project which consists of a number of sub-projects and that it could be
> beneficial more generally if you could more easily isolate out the
> sub-projects (not just for eglot, but possibly for other tools as well)
> at the project.el layer rather than the eglot layer. This could mean 
> the
> user has to be more involved in defining the project and sub-project
> structure/relationships, but that may be reasonable when you have more
> complex project structure?
> 
> I guess my question here is whether the focus should be on enhancing
> project.el rather than modifying/enhancing eglot.el to handle this use
> case?

It seems to me that while project.el could acquire the notion of
sub-projects, the *meaning* of a sub-project would be entirely
specific to the tool which needed it (eglot in this case).  And if you
had multiple tools which each wanted some kind of sub-project, you
might find that some of the sub-projects were overlapping others,
depending on the needs of the tools which each one was related to.

Still, if eglot could ask project.el for "the nearest sub-project
defined for 'eglot' usage, if any, and otherwise the main project" and
project.el had been told that for the project at /path/to/proj there
was an 'eglot' sub-project at /path/to/proj/subdir/foo, then that
could be useful.

So project.el could provide an API for defining and returning
sub-projects, but it would be up to eglot (or other tools) to cause
such sub-projects to have any kind of effect, and it would be up to
the end-user to define their 'eglot' sub-projects in the first place.

Re: Eglot, project.el, and python virtual environments

[Eli Zaretskii]

> Date: Fri, 18 Nov 2022 15:38:34 +1300
> From: Phil Sainty <psainty@orcon.net.nz>
> Cc: emacs-devel@gnu.org
> 
> > I guess my question here is whether the focus should be on enhancing
> > project.el rather than modifying/enhancing eglot.el to handle this use
> > case?
> 
> It seems to me that while project.el could acquire the notion of
> sub-projects, the *meaning* of a sub-project would be entirely
> specific to the tool which needed it (eglot in this case).  And if you
> had multiple tools which each wanted some kind of sub-project, you
> might find that some of the sub-projects were overlapping others,
> depending on the needs of the tools which each one was related to.
> 
> Still, if eglot could ask project.el for "the nearest sub-project
> defined for 'eglot' usage, if any, and otherwise the main project" and
> project.el had been told that for the project at /path/to/proj there
> was an 'eglot' sub-project at /path/to/proj/subdir/foo, then that
> could be useful.
> 
> So project.el could provide an API for defining and returning
> sub-projects, but it would be up to eglot (or other tools) to cause
> such sub-projects to have any kind of effect, and it would be up to
> the end-user to define their 'eglot' sub-projects in the first place.

I think this turns the table for no good reason.  I see no reason to
add complex new abstractions to project.el just because we have an
issue with configuring Eglot in the use case presented in this thread.

Let me remind you that Eglot already supports a kind of "sub-project":
it uses the same LSP server only for those source files in a project
that share the same major mode.  So parts of a project that use a
different PL are already considered to be a "sub-project", and Eglot
does that without any help from project.el.

Given that this feature already exists, a proposal to add a
"sub-project" notion to project.el should describe at least several
use cases of such "sub-projects" where the separate "sub-projects"
share the same programming language.  If the situation with python-env
is the only one we find reasonable, IMO adding "sub-projects" to
project.el is an unjustified complication.

I suggest to look at this as an Eglot issue, not a project.el issue.
What is requested here is an ability to tell Eglot which directories
should share the same LSP server and which ones should have separate
servers.  It shouldn't be hard to have a buffer-local variable to tell
Eglot that, or a function that accepts a buffer and returns a value
that Eglot can use for this decision.  All we need is a way to tell
Eglot which directories to communicate to the LSP server as those
which it should watch, and when to start another instance of the LSP
server even though one is already up and running for this project and
major mode.  Let's not complicate project.el for a problem that
doesn't belong to it.

Another evidence that this should be solved in Eglot is that "the
other LSP mode" doesn't depend on project for this.

I would also like to hear from Dmitry what are his thoughts on this.

Re: Eglot, project.el, and python virtual environments

[Danny Freeman]

Eli Zaretskii <eliz@gnu.org> writes:

> I think this turns the table for no good reason.  I see no reason to
> add complex new abstractions to project.el just because we have an
> issue with configuring Eglot in the use case presented in this thread.
>
> Let me remind you that Eglot already supports a kind of "sub-project":
> it uses the same LSP server only for those source files in a project
> that share the same major mode.  So parts of a project that use a
> different PL are already considered to be a "sub-project", and Eglot
> does that without any help from project.el.
>
> Given that this feature already exists, a proposal to add a
> "sub-project" notion to project.el should describe at least several
> use cases of such "sub-projects" where the separate "sub-projects"
> share the same programming language.  If the situation with python-env
> is the only one we find reasonable, IMO adding "sub-projects" to
> project.el is an unjustified complication.

I think that most monorepo projects fall into this category. That is a
large version controlled repository with multiple sub projects in it.
Sometimes the subprojects are written in different languages. Usually
there are sub folders of the monorepo project that act as sub projects.
I ran into one at work yesterday, but I'm not sure what I would have
project.el do differently there. I preferred it's behavior actually. 

> I suggest to look at this as an Eglot issue, not a project.el issue.
> What is requested here is an ability to tell Eglot which directories
> should share the same LSP server and which ones should have separate
> servers.  It shouldn't be hard to have a buffer-local variable to tell
> Eglot that, or a function that accepts a buffer and returns a value
> that Eglot can use for this decision.  All we need is a way to tell
> Eglot which directories to communicate to the LSP server as those
> which it should watch, and when to start another instance of the LSP
> server even though one is already up and running for this project and
> major mode.  Let's not complicate project.el for a problem that
> doesn't belong to it.

Is this not exactly what `eglot-lsp-context` is for? Using my example
from earlier in the thread is what I suspect is the "right way" to solve
this:

```
(defun project-find-virtualenv-for-eglot (dir)
  (when eglot-lsp-context
    (let ((root (locate-dominating-file dir ".virtualenv")))
      (when root
        (cons 'transient root)))))

(add-hook 'project-find-functions #'project-find-virtualenv-for-eglot)
```

It can be made more targeted by checking the value of directory if
necessary. (I could also a use when-let)

It is an obscure way to solve this problem. I only know about it from my
time spent with eglot's source. Not every user will have that
experience. How could we make that better?

> Another evidence that this should be solved in Eglot is that "the
> other LSP mode" doesn't depend on project for this.

The other lsp mode solves this by prompting the user for the root
directory where the lsp server should start and remembers the user's 
choice. I'm starting to understand why they did that: it handles a lot
of the cases like this where LSP root != project root, but I think we
can do better than a prompt. The majority of the projects I work on lsp
root is the same as the project root and I like not being bothered by a
prompt.

I would also be interested in João's thoughts on how this could be
handled. Copying him.

-- 
Danny Freeman

Re: Eglot, project.el, and python virtual environments

[Stefan Monnier]

> Let me remind you that Eglot already supports a kind of "sub-project":
> it uses the same LSP server only for those source files in a project
> that share the same major mode.

IIUC this is not always the right thing to do (for some multilanguage
projects where the LSP server supports similarly multiple languages).


        Stefan

Re: Eglot, project.el, and python virtual environments

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Phil Sainty <psainty@orcon.net.nz>,  Dmitry Gutov <dgutov@yandex.ru>,
>   theophilusx@gmail.com,  emacs-devel@gnu.org
> Date: Fri, 18 Nov 2022 10:06:49 -0500
> 
> > Let me remind you that Eglot already supports a kind of "sub-project":
> > it uses the same LSP server only for those source files in a project
> > that share the same major mode.
> 
> IIUC this is not always the right thing to do (for some multilanguage
> projects where the LSP server supports similarly multiple languages).

I didn't say it was _always_ TRT to do.  The question is: how
frequently is this TRT?  If it's a common situation, then the
exceptional situations need a specialized solution, but that solution
doesn't need to be in project.el.

Re: Eglot, project.el, and python virtual environments

[Eli Zaretskii]

> From: Danny Freeman <danny@dfreeman.email>
> Cc: Phil Sainty <psainty@orcon.net.nz>, Dmitry Gutov <dgutov@yandex.ru>,
>  theophilusx@gmail.com, emacs-devel@gnu.org, João Távora <joaotavora@gmail.com>
> Date: Fri, 18 Nov 2022 08:55:35 -0500
> 
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I think this turns the table for no good reason.  I see no reason to
> > add complex new abstractions to project.el just because we have an
> > issue with configuring Eglot in the use case presented in this thread.
> >
> > Let me remind you that Eglot already supports a kind of "sub-project":
> > it uses the same LSP server only for those source files in a project
> > that share the same major mode.  So parts of a project that use a
> > different PL are already considered to be a "sub-project", and Eglot
> > does that without any help from project.el.
> >
> > Given that this feature already exists, a proposal to add a
> > "sub-project" notion to project.el should describe at least several
> > use cases of such "sub-projects" where the separate "sub-projects"
> > share the same programming language.  If the situation with python-env
> > is the only one we find reasonable, IMO adding "sub-projects" to
> > project.el is an unjustified complication.
> 
> I think that most monorepo projects fall into this category. That is a
> large version controlled repository with multiple sub projects in it.

"Most"?  Maybe with Python projects (where I still doubt this
assertion, but have no experience to know for sure).  But otherwise
that is definitely not true.  Or else why would project.el decide to
consider a repository a single project?

IME, this kind of projects is a minority.

Re: Eglot, project.el, and python virtual environments

[Stefan Monnier]

>> IIUC this is not always the right thing to do (for some multilanguage
>> projects where the LSP server supports similarly multiple languages).
> I didn't say it was _always_ TRT to do.

No, indeed, I think it's a great default.  I'm just mentioning another
case where we need to override the default (and in that case, not to
split the project into subprojects but rather to "combine projects"
together), in the hope that it gives a better insight into what a good
general solution should look like.


        Stefan

Re: Eglot, project.el, and python virtual environments

[Danny Freeman]

Eli Zaretskii <eliz@gnu.org> writes:

>> I think that most monorepo projects fall into this category. That is a
>> large version controlled repository with multiple sub projects in it.
>
> "Most"?  Maybe with Python projects (where I still doubt this
> assertion, but have no experience to know for sure).  But otherwise
> that is definitely not true.  Or else why would project.el decide to
> consider a repository a single project?
>
> IME, this kind of projects is a minority.

"most monorepo" projects is what I'm saying, not most projects or
repositories in general. A monorepo is a repository with many
subprojects by definition. I agree they are in a minority, but they
exist. I'm just providing an example that was called for above in the
thread.

-- 
Danny Freeman

Re: Eglot, project.el, and python virtual environments

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 4396 bytes --]

Fwiw, Danny's tip is what I used at a recent day job gig for defining
subprojects in a 400k files repo.

João

On Wed, Nov 16, 2022, 22:39 Danny Freeman <danny@dfreeman.email> wrote:

>
> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
> > Hi,
> >
> > Here's another issue that's technically a emacs.help question, but might
> > result in some code/documentation updates, so I'm sending it here.
> >
> > My main day-job code base is an AWS CloudFormation monstrosity involving
> > several Python Lambdas, among other things. Basic project structure
> > looks like:
> >
> > project_root
> > ├── .git
> > ├── src
> > │   └── python
> > │       ├── VeryImportantLambda
> > │       │   └── .venv
> > │       ├── MoreImportance
> > │       │   └── .venv
> > │       ├── RunInCaseOfEmergency
> > │       │   └── .venv
> >
> > I'm using the python-lsp-server python package in each Python
> > subdirectory, and the key is that each of those directories is a virtual
> > environment that needs to stay isolated from the others. Each has
> > different packages installed, and in some cases even the Python versions
> > are different (though I'm trying to get rid of that).
> >
> > When I was using lsp-mode this wasn't difficult, because lsp-mode and
> > project.el are essentially orthogonal: if I visited a python file in a
> > given lambda directory, I could use `pyvenv-activate' to activate that
> > environment, and then the `lsp' invocation would confine itself to
> > python files within the environment. Project.el just provided
> > project-wide navigation.
> >
> > Now I'm trying to move to Eglot, and there is tighter integration
> > between Eglot and project.el. Turning on Eglot in one lambda starts the
> > server for all Python libraries in the whole project, not just the
> > current environment. I looked into constructing my own version of the
> > call to `eglot', but it is tightly tied to a project, all the way down.
> >
> > Is anyone else handling this situation? Any suggestions how to make it
> > work?
> >
> > Thanks,
> > Eric
>
>
> I have NOT been in this situation, but it sounds like you want to keep
> using project.el as is, but override eglot's usage of project.el to
> identify root directory to start a lsp server in.
>
> I think the place to start looking would be the function
> `eglot--current-project`.
>
> It uses a var called `eglot-lsp-context` which is `t` when eglot is
> searching for a project.
>
> Knowing this, there is another var called `project-find-functions` that
> project.el uses when searching for project roots, which can be used for
> finding custom project roots. I have one that looks for a file named
> ".project.el". If that file exists then that directory is identified as
> a project. It's really useful when you have a project not in source
> control.
>
> See:
> ```
> (defun project-find-project.el (dir)
>   "Returns a `manual-project' instance if the project of the current
> DIR has a .project.el file in its root directory."
>   (let ((root (locate-dominating-file dir ".project.el")))
>     (when root
>       (cons 'transient root))))
>
> (add-hook 'project-find-functions #'project-find-project.el)
> ```
>
> Now, I am not a python programmer, but lets pretend that python
> virtualenviroment depends on a file located in the directory you want to
> activate one of these virtual environments, say `.virtualenv`.
>
> You can write a project-find-functions implementation that looks for
> these virutalenv files ONLY when eglot-lsp-context is active.
>
> ```
> (defun project-find-virtualenv-for-eglot (dir)
>   (when eglot-lsp-context
>     (let ((root (locate-dominating-file dir ".virtualenv")))
>       (when root
>         (cons 'transient root)))))
>
> (add-hook 'project-find-functions #'project-find-virtualenv-for-eglot)
> ```
>
> I did not test this, but I think it should send you down the right path.
> Eglot should see your aws lambda folders as the project root, and
> project.el should see the parent.
>
> If python virtual environments do not have any kind of file marker in
> the filesystem, you could use a dummy file in those directories like
> `.eglot-project` or something.
>
> Hope this helps,
> --
> Danny Freeman
>
>

[-- Attachment #2: Type: text/html, Size: 5272 bytes --]

Re: Eglot, project.el, and python virtual environments

[Eric Abrahamsen]

Danny Freeman <danny@dfreeman.email> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> I think this turns the table for no good reason.  I see no reason to
>> add complex new abstractions to project.el just because we have an
>> issue with configuring Eglot in the use case presented in this thread.
>>
>> Let me remind you that Eglot already supports a kind of "sub-project":
>> it uses the same LSP server only for those source files in a project
>> that share the same major mode.  So parts of a project that use a
>> different PL are already considered to be a "sub-project", and Eglot
>> does that without any help from project.el.
>>
>> Given that this feature already exists, a proposal to add a
>> "sub-project" notion to project.el should describe at least several
>> use cases of such "sub-projects" where the separate "sub-projects"
>> share the same programming language.  If the situation with python-env
>> is the only one we find reasonable, IMO adding "sub-projects" to
>> project.el is an unjustified complication.
>
> I think that most monorepo projects fall into this category. That is a
> large version controlled repository with multiple sub projects in it.
> Sometimes the subprojects are written in different languages. Usually
> there are sub folders of the monorepo project that act as sub projects.
> I ran into one at work yesterday, but I'm not sure what I would have
> project.el do differently there. I preferred it's behavior actually. 
>
>> I suggest to look at this as an Eglot issue, not a project.el issue.
>> What is requested here is an ability to tell Eglot which directories
>> should share the same LSP server and which ones should have separate
>> servers.  It shouldn't be hard to have a buffer-local variable to tell
>> Eglot that, or a function that accepts a buffer and returns a value
>> that Eglot can use for this decision.  All we need is a way to tell
>> Eglot which directories to communicate to the LSP server as those
>> which it should watch, and when to start another instance of the LSP
>> server even though one is already up and running for this project and
>> major mode.  Let's not complicate project.el for a problem that
>> doesn't belong to it.
>
> Is this not exactly what `eglot-lsp-context` is for? Using my example
> from earlier in the thread is what I suspect is the "right way" to solve
> this:
>
> ```
> (defun project-find-virtualenv-for-eglot (dir)
>   (when eglot-lsp-context
>     (let ((root (locate-dominating-file dir ".virtualenv")))
>       (when root
>         (cons 'transient root)))))
>
> (add-hook 'project-find-functions #'project-find-virtualenv-for-eglot)
> ```
>
> It can be made more targeted by checking the value of directory if
> necessary. (I could also a use when-let)
>
> It is an obscure way to solve this problem. I only know about it from my
> time spent with eglot's source. Not every user will have that
> experience. How could we make that better?

I agree that this seems like the most likely point of customization --
and also agree that if this approach gets "blessed" somehow it could
probably be made more obvious, if only through documentation.

I wonder if the same entry point could be used for language server
short-circuiting: for instance, the transient project structure could
contain a sexp that's used to mask the value of `eglot-server-programs',
to point Eglot at different programs. It looks like that value would get
cached for later use.

Anyway, I haven't looked at Eglot closely enough to suggest anything
more than that. I'll try to read through the library this weekend.

Eric

Re: Eglot, project.el, and python virtual environments

[Dmitry Gutov]

On 18.11.2022 09:43, Eli Zaretskii wrote:
> Another evidence that this should be solved in Eglot is that "the
> other LSP mode" doesn't depend on project for this.

FWIW, "the other LSP mode" just provides more choices for the user in 
general, which is in line with its overall design.

> I would also like to hear from Dmitry what are his thoughts on this.

That's my inclination as well, based on the requested behavior in this 
thread.

We do have an old feature request in bug#54228 (which I'm hoping to 
resolve soon enough; more voices in that discussion would be welcome, by 
the way), but it can only be an answer here if people are okay with the 
"subprojects" behaving like separate projects altogether (meaning, for 
example, that project-find-file only offers files from the current 
subproject to jump).

Or else, we'd have a new notion of subprojects, and a separate set of 
commands to navigate/search/replace/etc in such. Not sure if that's what 
people want either.

And if not (to both), the proposed fixes using project-find-functions 
are not a good fit too.

Then I suppose Eglot could just learn a set of markers of its own (e.g. 
files named .eglot and/or build file names of popular tools). Not sure 
if LSP recommends something in that regard, or just defers to the 
editors. I wonder how VS Code and NeoVim, etc, make this decision.

Re: Eglot, project.el, and python virtual environments

[Dmitry Gutov]

On 18.11.2022 20:31, João Távora wrote:
> Fwiw, Danny's tip is what I used at a recent day job gig for defining 
> subprojects in a 400k files repo.

How did you like the drop in performance of 'M-x project-find-file' 
resulting from it?

Re: Eglot, project.el, and python virtual environments

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 681 bytes --]

I didn't even measure it because it never finished :-) I built
a backend-style based completion table based on
voidtools everything (https://www.voidtools.com/downloads/)
with regexp completion.  It's pretty fast.

I had hoped to plug it in project.el but I couldn't find a way, so
I rebound C-x p f to it.

João

On Sat, Nov 19, 2022 at 1:13 AM Dmitry Gutov <dgutov@yandex.ru> wrote:

> On 18.11.2022 20:31, João Távora wrote:
> > Fwiw, Danny's tip is what I used at a recent day job gig for defining
> > subprojects in a 400k files repo.
>
> How did you like the drop in performance of 'M-x project-find-file'
> resulting from it?
>


-- 
João Távora

[-- Attachment #2: Type: text/html, Size: 1212 bytes --]

Re: Eglot, project.el, and python virtual environments

[Eli Zaretskii]

> Date: Sat, 19 Nov 2022 03:12:21 +0200
> Cc: theophilusx@gmail.com, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> > I would also like to hear from Dmitry what are his thoughts on this.
> 
> That's my inclination as well, based on the requested behavior in this 
> thread.

What is your inclination?  I don't think I understand what does "that"
mean here.

Re: Eglot, project.el, and python virtual environments

[Dmitry Gutov]

On 19.11.2022 09:42, Eli Zaretskii wrote:
>> Date: Sat, 19 Nov 2022 03:12:21 +0200
>> Cc:theophilusx@gmail.com,emacs-devel@gnu.org
>> From: Dmitry Gutov<dgutov@yandex.ru>
>>
>>> I would also like to hear from Dmitry what are his thoughts on this.
>> That's my inclination as well, based on the requested behavior in this
>> thread.
> What is your inclination?  I don't think I understand what does "that"
> mean here.

My opinion is that if the users (as I'm reading in this thread) only 
want to influence Eglot's behavior (WRT to subproject root detection), 
but keep the bulk of project-related stuff (most importantly, project 
commands) working as they do now, then that should be implemented in Eglot.

If, on the other hand, they will want to use the newly introduced 
"subproject roots" for more things... then that can bear further 
consideration.

Re: Eglot, project.el, and python virtual environments

[Eli Zaretskii]

> Date: Sat, 19 Nov 2022 15:06:17 +0200
> Cc: psainty@orcon.net.nz, theophilusx@gmail.com, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 19.11.2022 09:42, Eli Zaretskii wrote:
> >> Date: Sat, 19 Nov 2022 03:12:21 +0200
> >> Cc:theophilusx@gmail.com,emacs-devel@gnu.org
> >> From: Dmitry Gutov<dgutov@yandex.ru>
> >>
> >>> I would also like to hear from Dmitry what are his thoughts on this.
> >> That's my inclination as well, based on the requested behavior in this
> >> thread.
> > What is your inclination?  I don't think I understand what does "that"
> > mean here.
> 
> My opinion is that if the users (as I'm reading in this thread) only 
> want to influence Eglot's behavior (WRT to subproject root detection), 
> but keep the bulk of project-related stuff (most importantly, project 
> commands) working as they do now, then that should be implemented in Eglot.
> 
> If, on the other hand, they will want to use the newly introduced 
> "subproject roots" for more things... then that can bear further 
> consideration.

Ah, okay, thanks.  This basically matches my understanding of the
issue.

Re: Eglot, project.el, and python virtual environments

[Dmitry Gutov]

On 19.11.2022 03:56, João Távora wrote:
> I didn't even measure it because it never finished :-)

Too bad. On my machine in gecko-dev (200000 files)

   (project-files (project-current))

finishes in a little over 1 second with warm fs cache.

> I built
> a backend-style based completion table based on
> voidtools everything (https://www.voidtools.com/downloads/ 
> <https://www.voidtools.com/downloads/>)
> with regexp completion.  It's pretty fast.

That's Windows-only, right?

> I had hoped to plug it in project.el but I couldn't find a way, so
> I rebound C-x p f to it.

Should be easy enough: you create your own backend (rather than reusing 
'transient') and override the generic 'project-files'. Then add it to 
project-find-functions.

Not sure about regexp matching (maybe completion styles would help), and 
you'll need to figure out the "ignores" mechanism -- how you would 
specify them: just picking up from .gitignore, or having a buffer-local 
variable.

Anyway, this is the problem with the popular recommendation: just add a 
thing to 'project-find-functions'. It degrades performance and changes 
how project commands work too.

OTOH, the fact that many people seem to accept the consequences anyway 
is probably saying something as well.

Re: Eglot, project.el, and python virtual environments

[Danny Freeman]

Dmitry Gutov <dgutov@yandex.ru> writes:

> Anyway, this is the problem with the popular recommendation: just add a thing to
> 'project-find-functions'. It degrades performance and changes how project commands work too.

Does it really change how the project commands work? That is not what I
have seen. The special project-find-functions thing only returns a
value when eglot is calling `project-current`. This only happens from
one function, `eglot--current-project`, which is called infrequently.
All other times that function returns nil so project.el falls back to
`project-try-vc`.  

Outside of eglot's one call to `project-current` the special
`project-find-functions` thing is only checking a single var for 
truthyness and returning nil. I don't see how it could adversely impact
performance. 

For reference this is the code I'm talking about:
```
(defun project-find-virtualenv-for-eglot (dir)

  (when eglot-lsp-context ;; ALWAYS NIL, except when called from `eglot--curent-project`

    (let ((root (locate-dominating-file dir ".virtualenv")))
      (when root
        (cons 'transient root)))))

(add-hook 'project-find-functions #'project-find-virtualenv-for-eglot)
```

-- 
Danny Freeman

Re: Eglot, project.el, and python virtual environments

[Dmitry Gutov]

Ah, sorry, I missed that part:

On 19.11.2022 21:17, Danny Freeman wrote:
> For reference this is the code I'm talking about:
> ```
> (defun project-find-virtualenv-for-eglot (dir)
> 
>    (when eglot-lsp-context ;; ALWAYS NIL, except when called from `eglot--curent-project`

Then your solution should work okay, but it also means it belongs to the 
category of Eglot hacks (as opposed to project.el hacks).

Since this issue seems to be common enough, though, we should decide on 
a proper fix for it, too.

Re: Eglot, project.el, and python virtual environments

[Danny Freeman]

Dmitry Gutov <dgutov@yandex.ru> writes:

> Then your solution should work okay, but it also means it belongs to the category of Eglot hacks (as
> opposed to project.el hacks).

It is absolutely an Eglot hack :)

> Since this issue seems to be common enough, though, we should decide on a proper fix for it, too.

Maybe a description of the problem along with the example code could be
written into the eglot documentation? 

Another option might be to setup the project-find-functions in eglot for
users. Some kind of configuration would be necessary. Maybe a var with
the intention of setting it in .dir-locals.el, that would be used to
store the name of an eglot "project" root file.

```
(defvar eglot-root-marker nil
  "File name used to identify the root directory to start a LSP server
  in. When nil, the current project root is used." )

(defun eglot--find-lsp-root (dir)
  (when-let ((root (and eglot-root-marker
                        eglot-lsp-context
                        (locate-dominating-file dir eglot-root-marker))))
    (cons 'transient root)))

;; Eglot configures this somewhere, maybe when the mode is set up?
(add-hook 'project-find-functions #'eglot--find-lsp-root)
```

That makes the solutions to OP's problem setting `eglot-root-marker` to
".venv" in a .dir-locals.el file in their project. I think there is also
a better name for `eglot-root-marker` in this example.

-- 
Danny Freeman

Re: Eglot, project.el, and python virtual environments

[Stephen Leake]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

>  Basic project structure
> looks like:
>
> project_root
> ├── .git
> ├── src
> │   └── python
> │       ├── VeryImportantLambda
> │       │   └── .venv
> │       ├── MoreImportance
> │       │   └── .venv
> │       ├── RunInCaseOfEmergency
> │       │   └── .venv
>
>
> Now I'm trying to move to Eglot, and there is tighter integration
> between Eglot and project.el. Turning on Eglot in one lambda starts the
> server for all Python libraries in the whole project, not just the
> current environment. I looked into constructing my own version of the
> call to `eglot', but it is tightly tied to a project, all the way down.
>
> Is anyone else handling this situation? Any suggestions how to make it
> work?

Declare a project.el project for each directory that has a .venv.

I'm guessing you are currently using vc-project, so there is only one,
rooted where the .git directory is.

You do this by writing your own function for project-find-functions;
something like:

(defun my-projects (dir)
 (list 'transient (locate-dominating-file ".venv")))

(add-to-list 'project-find-functions #'my-projects)

You might have to delete vc projects from project-find-functions, or
make it buffer-local; there are lots of choices.

-- 
-- Stephe

Re: Eglot, project.el, and python virtual environments

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 997 bytes --]

> > I built
> > a backend-style based completion table based on
> > voidtools everything (https://www.voidtools.com/downloads/
> > <https://www.voidtools.com/downloads/>)
> > with regexp completion.  It's pretty fast.
>
> That's Windows-only, right?
>

Yes. That's what I'm using there.

> I had hoped to plug it in project.el but I couldn't find a way, so
> > I rebound C-x p f to it.
>
> Should be easy enough: you create your own backend (rather than reusing
> 'transient') and override the generic 'project-files'. Then add it to
> project-find-functions.
>

iiuc project-files should return a list of files, or can it return a
completion table as well? If not the latter, then i think it won't work...

Anyway, this is the problem with the popular recommendation: just add a
> thing to 'project-find-functions'. It degrades performance and changes
> how project commands work too.
>

I don't see the relation. If anything this reduces the amount of files to
search (but it's still a lot).

>

[-- Attachment #2: Type: text/html, Size: 2337 bytes --]

Re: Eglot, project.el, and python virtual environments

[Dmitry Gutov]

On 20.11.2022 01:25, João Távora wrote:

>      > I had hoped to plug it in project.el but I couldn't find a way, so
>      > I rebound C-x p f to it.
> 
>     Should be easy enough: you create your own backend (rather than reusing
>     'transient') and override the generic 'project-files'. Then add it to
>     project-find-functions.
> 
> 
> iiuc project-files should return a list of files, or can it return a 
> completion table as well? If not the latter, then i think it won't work...

Yes, sorry for the confusion. I'm not sure how feasible it will be to 
use a completion table instead, it's something to look into. E.g. it 
would need to perform the conversion in project--read-file-cpd-relative 
lazily.

In my usage, though, project-find-file shows all available files right 
away, so it's a question whether the lazy computation will actually 
speed things up. Maybe if you require non-empty inputs only? And two 
chars or more? But what happens when you backspace and type again? The 
indexer rescans the directory? Unless such scans use a very fast index, 
just reading the whole list of files into memory can be faster.

>     Anyway, this is the problem with the popular recommendation: just add a
>     thing to 'project-find-functions'. It degrades performance and changes
>     how project commands work too.
> 
> 
> I don't see the relation. If anything this reduces the amount of files 
> to search (but it's still a lot).

It switches to a less efficient indexer ('find'), and it drops the 
handling of .gitignore. So, listing all files in a 100'000 file 
subproject might take longer than listing all files in the parent 
400'000 file project using 'git ls-files'.

Re: Eglot, project.el, and python virtual environments

[João Távora]

Danny Freeman <danny@dfreeman.email> writes:

> Dmitry Gutov <dgutov@yandex.ru> writes:
>
>> Then your solution should work okay, but it also means it belongs to the category of Eglot hacks (as
>> opposed to project.el hacks).
>
> It is absolutely an Eglot hack :)

I don't consider this a hack at all.  Not only does it work okay, it's
how supposed to work.

Dmitry has often suggested an Eglot-specific project.el backend, i.e. a
new type of object to specialize project.el generic functions to, but I
think that a good idea: Eglot is not a source of truth for project
information, like .git directories.  Instead Eglot is a _client_ for
that information as gathered from via some arbitrary method that Eglot
is angnostic to.  

The information is forwarded to the LSP server as the "workspace", but
the user needn't ever hear that M$ lingo.  So I don't think we should
invent an second notion of "LSP Workspace" into Emacs via Eglot.
Personally, when I tried lsp-mode a long time ago, "workspaces" was a
confusing concept and useless to me as a user.  Emacs users should just
think of "projects".

Most of the time, the default project-finding methods bundled with Emacs
work extremely well, in fact so well that it seems almost offensive when
they don't.  But it's crucial to recognize that this is not just because
of LSP reasons.  For example, simply because of the relatively poor
performance of the default project.el VC backend in a gargantuan repo of
400k files, I've had to define a different notion of "project" recently.

What constitutes a useful "project" for a user's setup can indeed vary
immensely.  So project.el should grow to address these corner cases,
maybe inventing new abstractions not tied to a particular client,
including LSP.  Maybe both this Python venv example and my "gargantuan
repo" example are hinting at possible uses for a "subproject"
abstraction?  Just food for thought.

> (defun eglot--find-lsp-root (dir)
>   (when-let ((root (and eglot-root-marker
>                         eglot-lsp-context
>                         (locate-dominating-file dir eglot-root-marker))))
>     (cons 'transient root)))
>
> (add-hook 'project-find-functions #'eglot--find-lsp-root)


I think the code above is fine, but it belongs in the user's
configuration.  In fact, it belongs in the manual, in a section similar
to what used to be section called "Handling Quirkly Servers" in the old
README.

This code doesn't belong in Eglot.  Eglot is designed as thin layer
making different Emacs facilities communicate with LSP servers.  Eglot's
user interaction is mostly for LSP basics such as connect/disconnect and
Eglot's user preferences deal with LSP in its generality.  Baking
project definition facilities into Eglot would bloat it and I don't want
that.

Are we worried that users will find it hard to apply snippets such as
Danny's?  My experience with libraries such as Yasnippet, SLY and Eglot
tells me otherwise: if robust simple Lisp snippets such as the one above
are easy to find, users have no trouble applying them.  But if we really
must have a point-and-click interactive interface for programming-averse
programmers :-) , then we can also remember that Eglot is also an
Elisp-level API for major-modes to take advantage of.

... Which means that if the problem here is both LSP and Python-specific
a .venv specific version of the above snippet could well live in
python.el.

Here, and in other cases such as complicated definitions of
eglot-server-programs, or making use of server-specific LSP methods,
there is no reason why a major-mode shouldn't (require 'eglot).

João

Re: Eglot, project.el, and python virtual environments

[Dmitry Gutov]

On 20.11.2022 03:51, João Távora wrote:
> Danny Freeman <danny@dfreeman.email> writes:
> 
>> Dmitry Gutov <dgutov@yandex.ru> writes:
>>
>>> Then your solution should work okay, but it also means it belongs to the category of Eglot hacks (as
>>> opposed to project.el hacks).
>>
>> It is absolutely an Eglot hack :)
> 
> I don't consider this a hack at all.  Not only does it work okay, it's
> how supposed to work.

It's a hack because it's a common case which every affected user ends up 
fixing through editing their init file.

> Most of the time, the default project-finding methods bundled with Emacs
> work extremely well, in fact so well that it seems almost offensive when
> they don't.  But it's crucial to recognize that this is not just because
> of LSP reasons.  For example, simply because of the relatively poor
> performance of the default project.el VC backend in a gargantuan repo of
> 400k files, I've had to define a different notion of "project" recently.

Or the OS used, with its different performance characteristics. And the 
performance of the W32 Emacs port as well. Anyway, this is beside the 
point, since we'll always want more improvements to performance, of 
course, to be able to handle larger projects on slower systems.

> What constitutes a useful "project" for a user's setup can indeed vary
> immensely.  So project.el should grow to address these corner cases,
> maybe inventing new abstractions not tied to a particular client,
> including LSP.  Maybe both this Python venv example and my "gargantuan
> repo" example are hinting at possible uses for a "subproject"
> abstraction?  Just food for thought.

If you want more food for thought, look up my previous messages in this 
thread regarding "subprojects". And others' too.

In this thread people are asking for a change in Eglot's behavior, and 
not in project-related commands. For example, to keep 
'project-find-file' listing all files in the parent project, not just in 
the current subproject.

Re: Eglot, project.el, and python virtual environments

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 333 bytes --]

On Sun, Nov 20, 2022, 01:49 João Távora <joaotavora@gmail.com> wrote:

Dmitry has often suggested an Eglot-specific project.el backend, i.e. a
> new type of object to specialize project.el generic functions to, but I
> think that a good idea:
>

Doh, obviously i meant to write "that is _not_ a good idea.

João

>
>

[-- Attachment #2: Type: text/html, Size: 908 bytes --]

Re: Eglot, project.el, and python virtual environments

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 20.11.2022 03:51, João Távora wrote:
>> Danny Freeman <danny@dfreeman.email> writes:
>> 
>>> Dmitry Gutov <dgutov@yandex.ru> writes:
>>>
>>>> Then your solution should work okay, but it also means it belongs to the category of Eglot hacks (as
>>>> opposed to project.el hacks).
>>>
>>> It is absolutely an Eglot hack :)
>> I don't consider this a hack at all.  Not only does it work okay,
>> it's
>> how supposed to work.
>
> It's a hack because it's a common case which every affected user ends
> up fixing through editing their init file.

So is every use of add-hook, for that matter.  Emacs has an init file
because it's not Microsoft Word.  The code you're describing is exactly
why I added the variable and how I intended it to work when I added 1.5
years ago.  That's as far as I'll wade into puerile arguments over
who/what is a "hack".

>> Most of the time, the default project-finding methods bundled with Emacs
>> work extremely well, in fact so well that it seems almost offensive when
>> they don't.  But it's crucial to recognize that this is not just because
>> of LSP reasons.  For example, simply because of the relatively poor
>> performance of the default project.el VC backend in a gargantuan repo of
>> 400k files, I've had to define a different notion of "project" recently.
>
> Or the OS used, with its different performance characteristics. And
> the performance of the W32 Emacs port as well. Anyway, this is beside
> the point, since we'll always want more improvements to performance,
> of course, to be able to handle larger projects on slower systems.

You shouldn't be writing performance off as a detail.  You can't just
wish it away, or think users will change OSs soon.  You should instead
think of solutions that help manage size and complexity.

But it's not just performance.  For example, in this particular project,
it makes sense, by default, to grep the superproject, but C-x f in the
subprojects.  Lack of subproject support in project.el means I have to
work around this with defadvice.

> If you want more food for thought, look up my previous messages in
> this thread regarding "subprojects". And others' too.

What about them?  I've said all I wanted to say on the matter.  If you
to make some specific point, make a specific point.

> In this thread people are asking for a change in Eglot's behavior, and
> not in project-related commands. For example, to keep
> 'project-find-file' listing all files in the parent project, not just
> in the current subproject.

People report problem they are facing.  They are free to suggest a real
or vapourware design to fix them.  My task, as Eglot maintainer, is to
understand the problems, consider the suggestions, and often point
people to real existing solutions.  In this case, Danny figured out
exactly how it's supposed to work from the docstrings of
eglot-lsp-context and eglot--current-project.  All that's needed is to
document it in the manual.  Alternatively, I've suggested that Danny's
code be added to python.el, so that the user need not change init files.

And you're of course free to make some egloss.el library that requires
eglot.el and has all the LSP paraphernalia you think is missing from
Eglot (maybe you'll end up with lsp-mode, who knows?).  But I think
better tools for configuring very large projects with of shapes and
sizes should be in the project management library of Emacs, which is
project.el.  menawhile, we have the real solutions described above.

João

Re: Eglot, project.el, and python virtual environments

[Dmitry Gutov]

On 20.11.2022 22:35, João Távora wrote:
> You shouldn't be writing performance off as a detail.  You can't just
> wish it away, or think users will change OSs soon.  You should instead
> think of solutions that help manage size and complexity.

No, I'm saying performance itself shouldn't sway the decision in this 
case one way or another. Other means to improve it exist.

> But it's not just performance.  For example, in this particular project,
> it makes sense, by default, to grep the superproject, but C-x f in the
> subprojects.  Lack of subproject support in project.el means I have to
> work around this with defadvice.

Perhaps in your particular project it makes sense. Most of the users I 
see in this thread seem to prefer it otherwise in their projects.

So that seems to indicate that the Eglot fix and the subprojects thing 
should be separate, implemented without tying one to the other.

If you have more things to say about the subprojects feature requests, 
BTW, please go ahead with 'M-x report-emacs-bug'. We'll need a more 
detailed description to decide whether to go ahead with it and how.

Re: Eglot, project.el, and python virtual environments

[Eric Abrahamsen]

João Távora <joaotavora@gmail.com> writes:

> Danny Freeman <danny@dfreeman.email> writes:
>
>> Dmitry Gutov <dgutov@yandex.ru> writes:
>>
>>> Then your solution should work okay, but it also means it belongs
>>> to the category of Eglot hacks (as
>>> opposed to project.el hacks).
>>
>> It is absolutely an Eglot hack :)
>
> I don't consider this a hack at all.  Not only does it work okay, it's
> how supposed to work.

[...]

>> (defun eglot--find-lsp-root (dir)
>>   (when-let ((root (and eglot-root-marker
>>                         eglot-lsp-context
>>                         (locate-dominating-file dir eglot-root-marker))))
>>     (cons 'transient root)))
>>
>> (add-hook 'project-find-functions #'eglot--find-lsp-root)
>
>
> I think the code above is fine, but it belongs in the user's
> configuration.  In fact, it belongs in the manual, in a section similar
> to what used to be section called "Handling Quirkly Servers" in the old
> README.
>
> This code doesn't belong in Eglot.  Eglot is designed as thin layer
> making different Emacs facilities communicate with LSP servers.  Eglot's
> user interaction is mostly for LSP basics such as connect/disconnect and
> Eglot's user preferences deal with LSP in its generality.  Baking
> project definition facilities into Eglot would bloat it and I don't want
> that.
>
> Are we worried that users will find it hard to apply snippets such as
> Danny's?  My experience with libraries such as Yasnippet, SLY and Eglot
> tells me otherwise: if robust simple Lisp snippets such as the one above
> are easy to find, users have no trouble applying them.  But if we really
> must have a point-and-click interactive interface for programming-averse
> programmers :-) , then we can also remember that Eglot is also an
> Elisp-level API for major-modes to take advantage of.
>
> ... Which means that if the problem here is both LSP and Python-specific
> a .venv specific version of the above snippet could well live in
> python.el.

From the point of view of an end-consumer of all this, I agree that
customization on my end is inevitable, and not a bad thing. I don't
expect that an automatic solution to Python virtual envs, in particular,
will ever be found: they are a mess, and there are several packages in
ELPA/MELPA just for dealing with them.

I spent some time looking at eglot.el and project.el, and was mostly
surprised to find how little they need to know about each other.
Really, Eglot only uses projects very lightly, to group managed buffers
together under one `eglot-lsp-server' instance (I don't know if the
actual external processes need to know which/how many buffers they're
involved with).

Other than that, it doesn't seem to matter at all that project.el and
eglot.el might have different opinions about what project any given file
buffer might belong to.

So yes, Danny's "hack" seems quite sufficient.

The only remaining question is finding the right LSP executable. In my
Python venv case, that's done by using one of the aforementioned
packages -- there's no guarantee that the virtual environment will be
under "./.venv", it could be somewhere else entirely. The packages
generally work by hacking exec-path, so that Eglot's `executable-find'
call finds the right Python LSP executable.

A little care needs to be taken that the environment is "activated"
before the call to `eglot', but it's not that tricky. I don't think
anything more should be done to help with this particular Python case
(besides some documentation).

But maybe something could be done for finding LSP executables in the
more general case. I can imagine that for saner languages, simply
customizing a different executable path (perhaps passing it different
command-line arguments) would be sufficient. All I could think of was
the possibility that the project structure returned by our custom
project-finding function could contain information about the LSP
executable. So instead of just

(cons 'transient dir)

We could return something like

(cons 'transient dir
      'eglot-lsp-program-overrides
         '((python-mode . ("~/proj/.venv/bin/python/pylsp"))))

Where the provided sexp is used as a mask on `eglot-server-programs'.

project.el is pretty vague about the structure of a project, so it's
hard to know exactly how that value should be injected. But with this
setup the user could either use 'transient directly, or first call some
other project-finding function, and then figure out how to stick this
information in there in a useful way. Then `eglot--guess-contact' could
check for it there.

Just a suggestion!

Thanks to all for the discussion,
Eric

Re: Eglot, project.el, and python virtual environments

[João Távora]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> I spent some time looking at eglot.el and project.el, and was mostly
> surprised to find how little they need to know about each other.

In particular, project.el knows nothing of eglot.el :-), as it should.

> So yes, Danny's "hack" seems quite sufficient.

Yes.  I will soon update the manual to contain 

> Just a suggestion!

Thanks, but have you tried setting a dir-local value for
eglot-server-programs?

João

Re: Eglot, project.el, and python virtual environments

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 20.11.2022 22:35, João Távora wrote:
>> You shouldn't be writing performance off as a detail.  You can't just
>> wish it away, or think users will change OSs soon.  You should instead
>> think of solutions that help manage size and complexity.
>
> No, I'm saying performance itself shouldn't sway the decision in this
> case one way or another.

I think I couldn't disagree more.  If there something that should
influence system design (usually as early as possible) are performance
considerations: they can't be an afterthought. That said, looking
forward to these "other means".

>> But it's not just performance.  For example, in this particular project,
>> it makes sense, by default, to grep the superproject, but C-x f in the
>> subprojects.  Lack of subproject support in project.el means I have to
>> work around this with defadvice.
>
> Perhaps in your particular project it makes sense. Most of the users I
> see in this thread seem to prefer it otherwise in their projects.
>
> So that seems to indicate that the Eglot fix and the subprojects thing
> should be separate, implemented without tying one to the other.

Certainly shouldn't be "tied" to the other, but if subproject
configuration becomes available in project.el, Eglot can easily take
advantage of it (perhaps even automatically).

As the Eglot "fix" for the current status quo, it's just a
documentation change to the manual.

João

Re: Eglot, project.el, and python virtual environments

[Alfred M. Szmidt]

   > I spent some time looking at eglot.el and project.el, and was mostly
   > surprised to find how little they need to know about each other.

   In particular, project.el knows nothing of eglot.el :-), as it should.

If eglot can make project.el better (or vice versa), then why not?
This seems like a short sighted absolutism.

Re: Eglot, project.el, and python virtual environments

[Eli Zaretskii]

> From: "Alfred M. Szmidt" <ams@gnu.org>
> Cc: eric@ericabrahamsen.net, danny@dfreeman.email, dgutov@yandex.ru,
>  emacs-devel@gnu.org
> Date: Mon, 21 Nov 2022 10:41:07 -0500
> 
>    > I spent some time looking at eglot.el and project.el, and was mostly
>    > surprised to find how little they need to know about each other.
> 
>    In particular, project.el knows nothing of eglot.el :-), as it should.
> 
> If eglot can make project.el better (or vice versa), then why not?

Because they are two almost orthogonal features.  It is good for them to be
only loosely coupled.

Re: Eglot, project.el, and python virtual environments

[Eric Abrahamsen]

João Távora <joaotavora@gmail.com> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> I spent some time looking at eglot.el and project.el, and was mostly
>> surprised to find how little they need to know about each other.
>
> In particular, project.el knows nothing of eglot.el :-), as it should.
>
>> So yes, Danny's "hack" seems quite sufficient.
>
> Yes.  I will soon update the manual to contain 

Thanks!

>> Just a suggestion!
>
> Thanks, but have you tried setting a dir-local value for
> eglot-server-programs?

I haven't, but I'm sure that would work fine. On the other hand, it
means more points of configuration (quite a few of them), and if we're
already returning a manually-constructed project, it would be nice to be
able to use that for everything necessary. Projects are so loosely
defined, it seems we could get a lot of mileage out of returning a
custom project type that held Eglot-specific configuration. I understand
you're unenthusiastic about having Eglot fool with project-related
configuration, and I understand why, but it seems like this could be a
very thin wrapper that was very Eglot-specific.

Anyway I don't need to push it. My original problem is solved, I'm glad
the documentation is being expanded, and I appreciate being able to move
to Eglot!

Thanks,
Eric

Re: Eglot, project.el, and python virtual environments

[Augusto Stoffel]

On Sun, 20 Nov 2022 at 23:54, Eric Abrahamsen wrote:

> The only remaining question is finding the right LSP executable. In my
> Python venv case, that's done by using one of the aforementioned
> packages -- there's no guarantee that the virtual environment will be
> under "./.venv", it could be somewhere else entirely. The packages
> generally work by hacking exec-path, so that Eglot's `executable-find'
> call finds the right Python LSP executable.
>
> A little care needs to be taken that the environment is "activated"
> before the call to `eglot', but it's not that tricky.

Using my buffer-env package, I never have to worry about activating a
venv.  It happens automatically.

> I don't think anything more should be done to help with this
> particular Python case (besides some documentation).
>
> But maybe something could be done for finding LSP executables in the
> more general case.

I disagree that there should be some special scheme to find a LSP
executable.  Note for instance that python.el has some ad-hoc stuff to
choose the right interpreter for the REPL, and here we are discussing
the same kind of issue again.

What's really needed instead is a scheme to set up the environment of a
project.  There are quite a few package besides buffer-env for that, but
AFAIK nothing built-in.

> I can imagine that for saner languages, simply customizing a different
> executable path (perhaps passing it different command-line arguments)
> would be sufficient.

Python is sane in this regard, right?

Re: Eglot, project.el, and python virtual environments

[Augusto Stoffel]

On Sun, 20 Nov 2022 at 17:36, Dmitry Gutov wrote:

> If you want more food for thought, look up my previous messages in
> this thread regarding "subprojects". And others' too.

How about the much simpler situation where you want a subdirectory of a
Git repo to count as its own project, completely disregarding any other
directories?

I guess it would be nice to support this by just creating a special
marker file, but I couldn't figure out if it's supported.

Re: Eglot, project.el, and python virtual environments

[Dmitry Gutov]

On 21/11/22 15:45, João Távora wrote:
> Dmitry Gutov <dgutov@yandex.ru> writes:
> 
>> On 20.11.2022 22:35, João Távora wrote:
>>> You shouldn't be writing performance off as a detail.  You can't just
>>> wish it away, or think users will change OSs soon.  You should instead
>>> think of solutions that help manage size and complexity.
>>
>> No, I'm saying performance itself shouldn't sway the decision in this
>> case one way or another.
> 
> I think I couldn't disagree more.  If there something that should
> influence system design (usually as early as possible) are performance
> considerations: they can't be an afterthought. That said, looking
> forward to these "other means".

Performance is important, but desired behavior comes first.

>>> But it's not just performance.  For example, in this particular project,
>>> it makes sense, by default, to grep the superproject, but C-x f in the
>>> subprojects.  Lack of subproject support in project.el means I have to
>>> work around this with defadvice.
>>
>> Perhaps in your particular project it makes sense. Most of the users I
>> see in this thread seem to prefer it otherwise in their projects.
>>
>> So that seems to indicate that the Eglot fix and the subprojects thing
>> should be separate, implemented without tying one to the other.
> 
> Certainly shouldn't be "tied" to the other, but if subproject
> configuration becomes available in project.el, Eglot can easily take
> advantage of it (perhaps even automatically).

I cannot reasonably add a feature with only one known (and expected) 
consumer. Whether we call it subprojects, or an eglot-only backend, or etc.

FWIW, "subprojects" that I can imagine would probably be implemented as 
a new hook or a list of "project markers". As long as only Eglot uses 
it, it might as well live there.

> As the Eglot "fix" for the current status quo, it's just a
> documentation change to the manual.

I think it's a problem when a significant part of userbase need to make 
a change to their config (a specific, non-trivial one) to make Eglot 
functional in their projects.

I sympathize with not wanting to collect that info for every file type 
and their dog in Eglot, but it's not 100% obvious to me that the place 
for that info is in major modes.

Re: Eglot, project.el, and python virtual environments

[Stefan Monnier]

>> I think I couldn't disagree more.  If there something that should
>> influence system design (usually as early as possible) are performance
>> considerations: they can't be an afterthought. That said, looking
>> forward to these "other means".
> Performance is important, but desired behavior comes first.

Can we just agree that both are important and that it can be hard to
know beforehand which consideration should dominate?


        Stefan

Re: Eglot, project.el, and python virtual environments

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1011 bytes --]

On Tue, Nov 22, 2022 at 9:12 PM Stefan Monnier <monnier@iro.umontreal.ca>
wrote:


> Can we just agree that both are important and that it can be hard to
> know beforehand which consideration should dominate?
>

I'd agree to that. But this is not "beforehand", there are really large
projects
out there.  Given a tree a tree structure such as a file system that can
have
very many nodes, not having any means to take advantage of that structure
tree-ness (as project.el clearly doesn't: see the protocol of project-files
and
the lack of sub-projects) is going to be a hard limitation.  Monorepos
are really popular in many businesses and many of these are large and/or
getting larger.

It does makes sense to start simple, but ignoring scale rarely yields
the "desired behaviour" unless that behavior is waiting forever.

Example: Git "started simple" then grew sparse checkouts, shallow
clones, worktrees.  You don't _have_ to use these features, but when you
do need them, it's very good that they are there.

[-- Attachment #2: Type: text/html, Size: 1835 bytes --]

Re: Eglot, project.el, and python virtual environments

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1934 bytes --]

On Tue, Nov 22, 2022, 15:48 Dmitry Gutov <dgutov@yandex.ru> wrote:

I cannot reasonably add a feature with only one known (and expected)
> consumer. Whether we call it subprojects, or an eglot-only backend, or etc.
>

I've given you two other cases of non-Eglot consumption.  In fact, my
problem isn't really Eglot/LSP-motivated at all: AFAICT clangd.exe works
fine with that monster repo.

Rather it's the fact that the monorepo is clearly organized into loosely
coupled
sub-projects and very often (but not always) it makes sense to do
sub-project
wide file-finding and reference-finding.  No LSP/Eglotness involved at all.

As I explained earlier, it would also help if project-files weren't so
"flat list"
oriented and allowed generalized collections, such as my completion table
based on a very fast external indexer.

> As the Eglot "fix" for the current status quo, it's just a
> > documentation change to the manual.
>
> I think it's a problem when a significant part of userbase need to make
> a change to their config (a specific, non-trivial one) to make Eglot
> functional in their projects.
>

Depends what you call significant, and non-trivial. Eglot aims to
make the common stuff trivial, and the complex stuff possible.
80-20 rule.

Of course, as soon as you discover that a certain usage pattern
is falling to the "80" side of things, you make it easier for that case. If
that
case is Python "venvs", and those are indeed very common, then make
Python mode use Eglot's and project.el's API: it's what they are there
 for, I would presume.

I sympathize with not wanting to collect that info for every file type
> and their dog in Eglot, but it's not 100% obvious to me that the place
> for that info is in major modes.
>

"File types" are handled by the major mode abstraction in Emacs.
Eglot strives not to change that and I think this is a winning idea.

João

>

[-- Attachment #2: Type: text/html, Size: 3508 bytes --]

Re: Eglot, project.el, and python virtual environments

[Dmitry Gutov]

On 22/11/22 23:34, João Távora wrote:
> Given a tree a tree structure such as a file system that can have
> very many nodes, not having any means to take advantage of that structure
> tree-ness (as project.el clearly doesn't: see the protocol of 
> project-files and
> the lack of sub-projects) is going to be a hard limitation.  Monorepos
> are really popular in many businesses and many of these are large and/or
> getting larger.

Like I said: if you want sub-projects, go and write a proper feature 
request, with expected behavior, which commands are affected, which are not.

It seems obvious to me from this thread that a lot of people don't 
actually want sub-projects as a separate entity (based on the requested 
behavior, hi Stefan!), they just want Eglot to work with the structure 
of their project.

Re: Eglot, project.el, and python virtual environments

[Dmitry Gutov]

On 22/11/22 23:40, João Távora wrote:
> Rather it's the fact that the monorepo is clearly organized into loosely 
> coupled
> sub-projects and very often (but not always) it makes sense to do 
> sub-project
> wide file-finding and reference-finding.  No LSP/Eglotness involved at all.

Speaking of monorepos, it seems to me that the boundaries between 
subprojects (or just nested projects) might not be defined by 
language-based separation of app directories, but rather by 
organizational needs (e.g. how teams are split). Teams could be 
backend-frontend-some-other-end, or they could be full-stack too.

IOW, the "subproject" structure needed by Eglot and one that looks 
natural to developers might quite well be different.

> As I explained earlier, it would also help if project-files weren't so 
> "flat list"
> oriented and allowed generalized collections, such as my completion table
> based on a very fast external indexer.

That's a separate feature request.

Re: Eglot, project.el, and python virtual environments

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 22/11/22 23:34, João Távora wrote:
>> Given a tree a tree structure such as a file system that can have
>> very many nodes, not having any means to take advantage of that structure
>> tree-ness (as project.el clearly doesn't: see the protocol of
>> project-files and
>> the lack of sub-projects) is going to be a hard limitation.  Monorepos
>> are really popular in many businesses and many of these are large and/or
>> getting larger.
>
> Like I said: if you want sub-projects, go and write a proper feature
> request, with expected behavior, which commands are affected, which
> are not.

I've just described in the other thread that I would like to have
finding references and finding files to be able to operate on either
sub-projects or super-projects on demand.  This is the problem I'm
facing, and it's not new.  For example, there is only: find file in the
very large project, and find file in the current directory.  There is no
"find file in this section of the repo, which is a sub-project in
itself".

IMO, it's not "improper" to describe problems and use cases: in fact I
prefer that people describe over jumping to vapourware solutions.  But
if you're really looking for a suggestion as to _how_ to design it, I
suppose my problem would be well dealt with a negative prefix on C-x p g
and C-x p f giving me a choice of which project to operate on.  But that
is only one possibility: new commands are also acceptable.

As to how one defines sub-projects, I think having
project-find-functions be used to compose a list of projects (as opposed
to to stopping after finding one) would be a nice way.  It would be nice
if the elements of project-find-functions could be informed of the
projects found so far.  But keeping the "stop after first" behaviour and
having members return a more complex object is maybe also good.  I
really don't care much what you eventually choose, as long as I can
solve my problem.

> It seems obvious to me from this thread that a lot of people don't
> actually want sub-projects as a separate entity (based on the
> requested behavior, hi Stefan!), they just want Eglot to work with the
> structure of their project.

The current Eglot solution to start servers in sub-directories of
repositories has already been described (and apparently accepted).

João

Re: Eglot, project.el, and python virtual environments

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 22/11/22 23:40, João Távora wrote:
>> Rather it's the fact that the monorepo is clearly organized into
>> loosely coupled
>> sub-projects and very often (but not always) it makes sense to do
>> sub-project
>> wide file-finding and reference-finding.  No LSP/Eglotness involved at all.
>
> Speaking of monorepos, it seems to me that the boundaries between
> subprojects (or just nested projects) might not be defined by
> language-based separation of app directories, but rather by
> organizational needs (e.g. how teams are split). Teams could be
> backend-frontend-some-other-end, or they could be full-stack too.

That's right.  Indeed that's a very common case and makes a perfect case
for sub-projects: multiple loosely interrelated hierarchies of code
living inside a monorepo.

> IOW, the "subproject" structure needed by Eglot and one that looks
> natural to developers might quite well be different.

Eglot doesn't need any structure: it's some LSP servers who want to be
launched in a specific LSP workspace, which Eglot equates to a Emacs
project -- and that could be well sub-project.  I'd say very often the
LSP concept of "workspace" would match one of the logical team
sub-projects, but if they don't, there's a perfectly workable current
eglot-lsp-context solution.

>> As I explained earlier, it would also help if project-files weren't
>> so "flat list"
>> oriented and allowed generalized collections, such as my completion table
>> based on a very fast external indexer.
>
> That's a separate feature request.

Yes, it is.

João

"Backend completion style" as a first-class library. Re: Eglot, project.el, and python virtual environments

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1733 bytes --]

On Tue, Nov 22, 2022 at 9:34 PM João Távora <joaotavora@gmail.com> wrote:

> It does makes sense to start simple, but ignoring scale rarely yields
> the "desired behaviour" unless that behavior is waiting forever.
>

Speaking of "ignoring scale", I've been meaning to propose here that
the "Backend completion style" you once wrote, Stefan, be added
to Emacs and to GNU Elpa as a separate :core package.

This style is very useful: I've used it successfully in three separate
occasions:  Eglot, SLY and the voidtools everything file indexer
completion table (https://github.com/joaotavora/eel/blob/master/eel.el)

It's true that it stings a bit to bring in a completion style whose purpose
is to negate the use of all other fancy styles and let the external
backend do some unknown idiosyncratic pattern matching for us.
But it ends up working so well in practice...

As data sets grow larger, I think we have to come to terms that
Emacs is just not very performant in managing very large collections
of strings, and other special-purpose external tools implemented in other
languages are just faster, not to mention the cases where you _have_
to interact with those tools for other reasons anyway and you just can't
afford to transfer the whole dataset to Emacs as a big list of strings.

IOW, I think Emacs's styles are pretty good for its own Elisp Lisp
machine (until about a couple of hundred thousand symbols) and
for managing small collections, but for anything larger we need
other techniques.

So the idea is to create a `lisp/progmodes/backend-completion.lisp` with
the  generalized contents of what is now the ";;; Backend completion"
section of eglot.el.  WDYT, Stefan et al?

João

[-- Attachment #2: Type: text/html, Size: 2529 bytes --]

Re: Eglot, project.el, and python virtual environments

[Dmitry Gutov]

On 23/11/22 01:23, João Távora wrote:
> Dmitry Gutov <dgutov@yandex.ru> writes:
> 
>> On 22/11/22 23:34, João Távora wrote:
>>> Given a tree a tree structure such as a file system that can have
>>> very many nodes, not having any means to take advantage of that structure
>>> tree-ness (as project.el clearly doesn't: see the protocol of
>>> project-files and
>>> the lack of sub-projects) is going to be a hard limitation.  Monorepos
>>> are really popular in many businesses and many of these are large and/or
>>> getting larger.
>>
>> Like I said: if you want sub-projects, go and write a proper feature
>> request, with expected behavior, which commands are affected, which
>> are not.
> 
> I've just described in the other thread that I would like to have
> finding references and finding files to be able to operate on either
> sub-projects or super-projects on demand.  This is the problem I'm
> facing, and it's not new.  For example, there is only: find file in the
> very large project, and find file in the current directory.  There is no
> "find file in this section of the repo, which is a sub-project in
> itself".

It would be much more helpful in a dedicated bug report where we could 
discuss the details, collect the votes and see what kind of design will 
ultimately satisfy the requirements. Instead of drowning it all in this 
thread, which is only moderately related.

> IMO, it's not "improper" to describe problems and use cases: in fact I
> prefer that people describe over jumping to vapourware solutions.  But
> if you're really looking for a suggestion as to _how_ to design it, I
> suppose my problem would be well dealt with a negative prefix on C-x p g
> and C-x p f giving me a choice of which project to operate on.  But that
> is only one possibility: new commands are also acceptable.

Note that you can more-or-less do this now: press 'C-x p p', select the 
parent project from the list, then choose 'f' or 'g' to run the command 
there. So to justify the added complexity one should say that they do 
want to use this feature frequently enough to justify the added 
complexity (which will reduce the number of keystrokes).

And even if we do, we might not need the additional notion of 
sub-projects. E.g. 'search in the parent project (if any)' might work as 
"take the root, go up a directory, search for a project there; if it 
exists, use it". Though Stephen L. might want a generic for that, since 
his projects do not correlate with directory tree.

OTOH, if one of the operations will require a step "get all subprojects 
of the current project", then a separate notion might be required, with 
a separate hook.

> As to how one defines sub-projects, I think having
> project-find-functions be used to compose a list of projects (as opposed
> to to stopping after finding one) would be a nice way.  It would be nice
> if the elements of project-find-functions could be informed of the
> projects found so far.  But keeping the "stop after first" behaviour and
> having members return a more complex object is maybe also good.  I
> really don't care much what you eventually choose, as long as I can
> solve my problem.

If we wanted to define projects and subprojects through the same hook, 
the step "get all subprojects of the current project" might not be 
feasible to implement. Because of performance-related considerations.

Re: "Backend completion style" as a first-class library. Re: Eglot, project.el, and python virtual environments

[Stefan Monnier]

> Speaking of "ignoring scale", I've been meaning to propose here that
> the "Backend completion style" you once wrote, Stefan, be added
> to Emacs and to GNU Elpa as a separate :core package.

Please do.  I've been meaning to do that for quite a while but it never
reaches the top of my todo list.

> So the idea is to create a `lisp/progmodes/backend-completion.lisp` with
> the  generalized contents of what is now the ";;; Backend completion"
> section of eglot.el.  WDYT, Stefan et al?

Sounds about right, modulo the obvious s/.lisp/.el/


        Stefan

Re: Eglot, project.el, and python virtual environments

[Dmitry Gutov]

On 21/11/22 22:58, Augusto Stoffel wrote:
> On Sun, 20 Nov 2022 at 17:36, Dmitry Gutov wrote:
> 
>> If you want more food for thought, look up my previous messages in
>> this thread regarding "subprojects". And others' too.
> How about the much simpler situation where you want a subdirectory of a
> Git repo to count as its own project, completely disregarding any other
> directories?
> 
> I guess it would be nice to support this by just creating a special
> marker file, but I couldn't figure out if it's supported.

It's a seemingly simple issue which becomes hard once you want to create 
a turnkey solution for everybody which is simultaneously fast, easy to 
use and still flexible. Or at least for me it did.

There are a couple of patches I put up for discussion in 
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=41572#82, but alas the 
discussion that arose went into some other direction.

In short, the issues are:

- Whether we need to be able to exclude the subprojects from 
project-files of the parent project. Can be useful with large projects; 
not so easy to do if the subprojects are defined by some file markers 
down in the directory tree.

- If we don't, I guess nobody needs the defcustom 
'project-vc-merge-submodules' either?

- Or if we do, would it be okay to recommend the user customizes the 
ignores of the parent project manually to exclude the specific 
subprojects? In addition to setting up the markers. And this way we lose 
the potential ability to add their dirs to project-files of the parent, 
to make them more easy to reach (project-vc currently does that with 
project-vc-merge-submodules=nil), or list them through a separate 
command, etc.

- Would markers be limited to file names (perhaps with wilcards)? If so, 
it might be feasible after all to exclude marker-based subprojects if 
the user so wishes using an extra 'git ls-files -co -- ...' call. It's 
not free, though.

- Should subprojects obey "ignores" of the parent project? If they do 
(which would make sense in practice -- obeying .gitignore of the repo), 
how do we describe that semantically if a subproject is a separate 
project type/backend?

Anyway, I suggest you give the original messages a read (#82 and #83, at 
least) and let me know what you think. Preferably in the bug comments, 
since those will be easier to find later.

Subprojects in project.el (Was: Eglot, project.el, and python virtual environments)

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

>> I've just described in the other thread that I would like to have
>> finding references and finding files to be able to operate on either
>> sub-projects or super-projects on demand.  This is the problem I'm
>> facing, and it's not new.  For example, there is only: find file in the
>> very large project, and find file in the current directory.  There is no
>> "find file in this section of the repo, which is a sub-project in
>> itself".
>
> It would be much more helpful in a dedicated bug report where we could
> discuss the details, collect the votes and see what kind of design
> will ultimately satisfy the requirements. Instead of drowning it all
> in this thread, which is only moderately related.

I think we're doing fine here but I've changed the subject line to
"unbury" it from the thread.

>> IMO, it's not "improper" to describe problems and use cases: in fact I
>> prefer that people describe over jumping to vapourware solutions.  But
>> if you're really looking for a suggestion as to _how_ to design it, I
>> suppose my problem would be well dealt with a negative prefix on C-x p g
>> and C-x p f giving me a choice of which project to operate on.  But that
>> is only one possibility: new commands are also acceptable.
>
> Note that you can more-or-less do this now: press 'C-x p p', select
> the parent project from the list, then choose 'f' or 'g' to run the
> command there. So to justify the added complexity one should say that
> they do want to use this feature frequently enough to justify the
> added complexity (which will reduce the number of keystrokes).

Both cases -- focus on super-project/focus on sub-project -- are
definitely common.  I have no problem in using an existing interface,
even if it idiosyncratic.

So if the inner-most sub-project is the default but somehow I can
explicitly call up a "project picker" that shows me the super-project as
one of the first options, that probably works fine.

> And even if we do, we might not need the additional notion of
> sub-projects. E.g. 'search in the parent project (if any)' might work
> as "take the root, go up a directory, search for a project there; if
> it exists, use it". Though Stephen L. might want a generic for that,
> since his projects do not correlate with directory tree.

And in my case, sub-projects don't necessarily exist exactly one-level
down super-project's root.  Sometimes they do, but now always.

Of course, just as data point, I've been solving all my problems outside
project.el for a good long while now, even before I was faced with this
gigantic monorepo and even before project.el was a thing.  I use Leo
Liu's ack-el.el, which greps from the super-project's root by default,
and from wherever I want given enough C-u.  And I have a similar
separate command for finding files from arbitrary points in the
hierarchy.  And I have a similar command for compiling.  I just happen
to think having this in project.el would make a lot of sense.  I've
recently experimented with it and found the new project.el features
pleasant to work in smaller or well-behaved projects.  It's not powerful
enough for the big stuff, but it ought to be.

> OTOH, if one of the operations will require a step "get all
> subprojects of the current project", then a separate notion might be
> required, with a separate hook.
>
>> As to how one defines sub-projects, I think having
>> project-find-functions be used to compose a list of projects (as opposed
>> to to stopping after finding one) would be a nice way.  It would be nice
>> if the elements of project-find-functions could be informed of the
>> projects found so far.  But keeping the "stop after first" behaviour and
>> having members return a more complex object is maybe also good.  I
>> really don't care much what you eventually choose, as long as I can
>> solve my problem.
>
> If we wanted to define projects and subprojects through the same hook,
> the step "get all subprojects of the current project" might not be
> feasible to implement. Because of performance-related considerations.

Those remain to be seen, I can't guess what things you're imagining.

I'm not sure we need that step at all.  After all there is no "get me
all the projects in my hard drive" either, and at least I haven't missed
it.

Anyway, from my POV this type of sub-project definition would not be
very nice.

  (add-hook 'some-new-subproject-find-hook
            (lambda (dir)
              (cons 'super-project
                    (some-code-which-scans-and-returns-a-list-of-subprojects))))

I would prefer this, which I think is more readable and in-line with
current project.el:

  (add-hook 'project-find-functions
            (lambda (dir)
              (let ((super (project-current)))
                (when (and super
                           (string= (file-name-directory
                                     (directory-file-name dir))
                                    (project-root super))
                           (my-special-condition dir))
                  (cons 'transient dir)))))


Maybe 'my-special-condition' could read a directory local variable to
decide if sub-projects are enabled or not.

For my idea to work, project-current would have to be changed to

1. Call all members of project-find-functions: return the innermost
   project found thus far.
2. Not call project-find-functions recursively when called from within a function
   in project-find-functions, just return the innermost project found
   thus far.

I don't see any big performance problems in this alternative.

But, again, the above are suggestions: _any_ alternative that lets me
define inner sub-projects within a larger super-project and let me
choose where I want to grep, find-file, compile, would be an
improvement.

João

Re: Subprojects in project.el (Was: Eglot, project.el, and python virtual environments)

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 9222 bytes --]

Hi,

In the previous email, my pseudo-code was all botched, as usual when
writing imaginationware.  Better back up my ideas with real code.  The
patch after my sig allows these kinds of customizations:

  (defun joaot/sub-projects (dir)
    (let* ((super (project-current))
           (super-root (project-root super))
           (relative (file-relative-name dir super-root))
           (d (car (split-string relative "/"))))
      (when (member d '("lisp" "src"))
        (cons 'transient (expand-file-name (concat d "/") super-root)))))

  (add-hook 'project-find-functions
            'joaot/sub-projects
            t nil)

This will define two subprojects inside Emacs's tree.  The "lisp" and
"src" projects.  It's not particularly useful for Emacs, but I hope it
illustrates what I want to do.

Of course, I still haven't coded any commands to access the "super
project" so all commands work on the innermost sub-project, but they
do seem to work unproblematically.

To get the super-project, a seemingly robust way is:

  (project-current nil
    (file-name-directory
       (directory-file-name (project-root (project-current)))))

But I've yet to find where to plug it.  Maybe tomorrow I'll spend
some more time playing around with this.

João

This is the project.el patch:

diff --git a/lisp/progmodes/project.el b/lisp/progmodes/project.el
index c7b2c386ccd..1903b7f622f 100644
--- a/lisp/progmodes/project.el
+++ b/lisp/progmodes/project.el
@@ -196,13 +196,13 @@ project-current
 See the doc string of `project-find-functions' for the general form
 of the project instance object."
   (unless directory (setq directory default-directory))
-  (let ((pr (project--find-in-directory directory)))
+  (let ((pr (car (project--find-in-directory directory))))
     (cond
      (pr)
      ((unless project-current-inhibit-prompt
         maybe-prompt)
       (setq directory (project-prompt-project-dir)
-            pr (project--find-in-directory directory))))
+            pr (car (project--find-in-directory directory)))))
     (when maybe-prompt
       (if pr
           (project-remember-project pr)
@@ -211,8 +211,22 @@ project-current
         (setq pr (cons 'transient directory))))
     pr))

+(defvar project--overriding-projects nil)
+
 (defun project--find-in-directory (dir)
-  (run-hook-with-args-until-success 'project-find-functions dir))
+  "Find projects that directory DIR belongs to.
+Return them as a list.  If more than one is found, the first is
+usually the innermost."
+  (or project--overriding-projects
+      (let (projects)
+        (run-hook-wrapped
+         'project-find-functions
+         (lambda (fn)
+           (let* ((project--overriding-projects projects)
+                  (prj (funcall fn dir)))
+             (when prj (push prj projects)))
+           nil))
+        projects)))

 (defvar project--within-roots-fallback nil)

On Wed, Nov 23, 2022 at 1:56 PM João Távora <joaotavora@gmail.com> wrote:

> Dmitry Gutov <dgutov@yandex.ru> writes:
>
> >> I've just described in the other thread that I would like to have
> >> finding references and finding files to be able to operate on either
> >> sub-projects or super-projects on demand.  This is the problem I'm
> >> facing, and it's not new.  For example, there is only: find file in the
> >> very large project, and find file in the current directory.  There is no
> >> "find file in this section of the repo, which is a sub-project in
> >> itself".
> >
> > It would be much more helpful in a dedicated bug report where we could
> > discuss the details, collect the votes and see what kind of design
> > will ultimately satisfy the requirements. Instead of drowning it all
> > in this thread, which is only moderately related.
>
> I think we're doing fine here but I've changed the subject line to
> "unbury" it from the thread.
>
> >> IMO, it's not "improper" to describe problems and use cases: in fact I
> >> prefer that people describe over jumping to vapourware solutions.  But
> >> if you're really looking for a suggestion as to _how_ to design it, I
> >> suppose my problem would be well dealt with a negative prefix on C-x p g
> >> and C-x p f giving me a choice of which project to operate on.  But that
> >> is only one possibility: new commands are also acceptable.
> >
> > Note that you can more-or-less do this now: press 'C-x p p', select
> > the parent project from the list, then choose 'f' or 'g' to run the
> > command there. So to justify the added complexity one should say that
> > they do want to use this feature frequently enough to justify the
> > added complexity (which will reduce the number of keystrokes).
>
> Both cases -- focus on super-project/focus on sub-project -- are
> definitely common.  I have no problem in using an existing interface,
> even if it idiosyncratic.
>
> So if the inner-most sub-project is the default but somehow I can
> explicitly call up a "project picker" that shows me the super-project as
> one of the first options, that probably works fine.
>
> > And even if we do, we might not need the additional notion of
> > sub-projects. E.g. 'search in the parent project (if any)' might work
> > as "take the root, go up a directory, search for a project there; if
> > it exists, use it". Though Stephen L. might want a generic for that,
> > since his projects do not correlate with directory tree.
>
> And in my case, sub-projects don't necessarily exist exactly one-level
> down super-project's root.  Sometimes they do, but now always.
>
> Of course, just as data point, I've been solving all my problems outside
> project.el for a good long while now, even before I was faced with this
> gigantic monorepo and even before project.el was a thing.  I use Leo
> Liu's ack-el.el, which greps from the super-project's root by default,
> and from wherever I want given enough C-u.  And I have a similar
> separate command for finding files from arbitrary points in the
> hierarchy.  And I have a similar command for compiling.  I just happen
> to think having this in project.el would make a lot of sense.  I've
> recently experimented with it and found the new project.el features
> pleasant to work in smaller or well-behaved projects.  It's not powerful
> enough for the big stuff, but it ought to be.
>
> > OTOH, if one of the operations will require a step "get all
> > subprojects of the current project", then a separate notion might be
> > required, with a separate hook.
> >
> >> As to how one defines sub-projects, I think having
> >> project-find-functions be used to compose a list of projects (as opposed
> >> to to stopping after finding one) would be a nice way.  It would be nice
> >> if the elements of project-find-functions could be informed of the
> >> projects found so far.  But keeping the "stop after first" behaviour and
> >> having members return a more complex object is maybe also good.  I
> >> really don't care much what you eventually choose, as long as I can
> >> solve my problem.
> >
> > If we wanted to define projects and subprojects through the same hook,
> > the step "get all subprojects of the current project" might not be
> > feasible to implement. Because of performance-related considerations.
>
> Those remain to be seen, I can't guess what things you're imagining.
>
> I'm not sure we need that step at all.  After all there is no "get me
> all the projects in my hard drive" either, and at least I haven't missed
> it.
>
> Anyway, from my POV this type of sub-project definition would not be
> very nice.
>
>   (add-hook 'some-new-subproject-find-hook
>             (lambda (dir)
>               (cons 'super-project
>
> (some-code-which-scans-and-returns-a-list-of-subprojects))))
>
> I would prefer this, which I think is more readable and in-line with
> current project.el:
>
>   (add-hook 'project-find-functions
>             (lambda (dir)
>               (let ((super (project-current)))
>                 (when (and super
>                            (string= (file-name-directory
>                                      (directory-file-name dir))
>                                     (project-root super))
>                            (my-special-condition dir))
>                   (cons 'transient dir)))))
>
>
> Maybe 'my-special-condition' could read a directory local variable to
> decide if sub-projects are enabled or not.
>
> For my idea to work, project-current would have to be changed to
>
> 1. Call all members of project-find-functions: return the innermost
>    project found thus far.
> 2. Not call project-find-functions recursively when called from within a
> function
>    in project-find-functions, just return the innermost project found
>    thus far.
>
> I don't see any big performance problems in this alternative.
>
> But, again, the above are suggestions: _any_ alternative that lets me
> define inner sub-projects within a larger super-project and let me
> choose where I want to grep, find-file, compile, would be an
> improvement.
>
> João
>


-- 
João Távora

[-- Attachment #2: Type: text/html, Size: 11012 bytes --]

Re: Subprojects in project.el (Was: Eglot, project.el, and python virtual environments)

[Dmitry Gutov]

On 23/11/22 22:33, João Távora wrote:
> For my idea to work, project-current would have to be changed to
> 
> 1. Call all members of project-find-functions: return the innermost
>     project found thus far.
> 2. Not call project-find-functions recursively when called from within a 
> function
>     in project-find-functions, just return the innermost project found
>     thus far.
> 
> I don't see any big performance problems in this alternative.

Might not be much of a difference in a simple configuration, but going 
up and down dirs is not free even on a fast SSD, especially as the 
number of the elements on this hook grows.

But where you will definitely see a difference, is when using this with 
Tramp.

Re: Subprojects in project.el (Was: Eglot, project.el, and python virtual environments)

[Dmitry Gutov]

On 23/11/22 22:33, João Távora wrote:
>    (defun joaot/sub-projects (dir)
>      (let* ((super (project-current))
>             (super-root (project-root super))
>             (relative (file-relative-name dir super-root))
>             (d (car (split-string relative "/"))))
>        (when (member d '("lisp" "src"))
>          (cons 'transient (expand-file-name (concat d "/") super-root)))))
> 
>    (add-hook 'project-find-functions
>              'joaot/sub-projects
>              t nil)
> 
> This will define two subprojects inside Emacs's tree.  The "lisp" and
> "src" projects.  It's not particularly useful for Emacs, but I hope it
> illustrates what I want to do.

So you're fine with having to specify the list of subprojects for every 
project manually? Via a hoook or generic override.

As opposed to having a globally configured list of (sub)project markers 
which would do a good job 90% of the time.

Re: Subprojects in project.el (Was: Eglot, project.el, and python virtual environments)

[Dmitry Gutov]

On 23/11/22 15:57, João Távora wrote:
>> It would be much more helpful in a dedicated bug report where we could
>> discuss the details, collect the votes and see what kind of design
>> will ultimately satisfy the requirements. Instead of drowning it all
>> in this thread, which is only moderately related.
> I think we're doing fine here but I've changed the subject line to
> "unbury" it from the thread.

You're fussy about proper bug reporting in your projects, but somehow do 
not extend that courtesy to others.

>> And even if we do, we might not need the additional notion of
>> sub-projects. E.g. 'search in the parent project (if any)' might work
>> as "take the root, go up a directory, search for a project there; if
>> it exists, use it". Though Stephen L. might want a generic for that,
>> since his projects do not correlate with directory tree.
> And in my case, sub-projects don't necessarily exist exactly one-level
> down super-project's root.  Sometimes they do, but now always.

I don't think that's important whether it's one-level down exactly:

(defun project-parent (project)
   (project-current nil (file-name-as-directory
                         (file-name-directory
                          (directory-file-name (project-root project))))))

>> If we wanted to define projects and subprojects through the same hook,
>> the step "get all subprojects of the current project" might not be
>> feasible to implement. Because of performance-related considerations.
> Those remain to be seen, I can't guess what things you're imagining.

I'm imagining that traversing a directory tree with an arbitrary 
predicate is going to be slow. If the predicate is limited somehow (e.g. 
to a list of "markers" as base file name, or at least wildcards), 'git 
ls-files' can probably handle this, with certain but bounded cost.

I wrote about this in another email here yesterday.

Re: Subprojects in project.el (Was: Eglot, project.el, and python virtual environments)

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Wed, 23 Nov 2022 20:33:04 +0000
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,
>  Danny Freeman <danny@dfreeman.email>, 
>  Eric Abrahamsen <eric@ericabrahamsen.net>, emacs-devel <emacs-devel@gnu.org>
> 
>   (defun joaot/sub-projects (dir)
>     (let* ((super (project-current))
>            (super-root (project-root super))
>            (relative (file-relative-name dir super-root))
>            (d (car (split-string relative "/"))))
>       (when (member d '("lisp" "src"))
>         (cons 'transient (expand-file-name (concat d "/") super-root)))))

Is this okay for code outside project.el to know the internal structure of a
project object in general, and in particular cons "by hand" a 'transient'
project object?  I thought this was a closely-guarded implementation detail
that even doc strings are not allowed to divulge?

Re: Subprojects in project.el (Was: Eglot, project.el, and python virtual environments)

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1191 bytes --]

On Thu, Nov 24, 2022 at 2:49 AM Dmitry Gutov <dgutov@yandex.ru> wrote:

> On 23/11/22 22:33, João Távora wrote:
> > For my idea to work, project-current would have to be changed to
> >
> > 1. Call all members of project-find-functions: return the innermost
> >     project found thus far.
> > 2. Not call project-find-functions recursively when called from within a
> > function
> >     in project-find-functions, just return the innermost project found
> >     thus far.
> >
> > I don't see any big performance problems in this alternative.
>
> Might not be much of a difference in a simple configuration, but going
> up and down dirs is not free even on a fast SSD, especially as the
> number of the elements on this hook grows.
>
> But where you will definitely see a difference, is when using this with
> Tramp.
>

Have you seen the code I posted?  You'll notice that it doesn't do
any file system operations, it just does file name operations. And
those run in CPU/memory, not file system. So I cannot understand
your argument.  But I am happy to measure performance in different
scenarios, if you'd like to suggest a particularly problematic one.

João

[-- Attachment #2: Type: text/html, Size: 1627 bytes --]

Re: Subprojects in project.el (Was: Eglot, project.el, and python virtual environments)

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1794 bytes --]

On Thu, Nov 24, 2022 at 3:01 AM Dmitry Gutov <dgutov@yandex.ru> wrote:

> On 23/11/22 15:57, João Távora wrote:
> >> It would be much more helpful in a dedicated bug report where we could
> >> discuss the details, collect the votes and see what kind of design
> >> will ultimately satisfy the requirements. Instead of drowning it all
> >> in this thread, which is only moderately related.
> > I think we're doing fine here but I've changed the subject line to
> > "unbury" it from the thread.
>
> You're fussy about proper bug reporting in your projects, but somehow do
> not extend that courtesy to others.
>

I am indeed "fussy" about "bug reporting".  But here, Dmitry, I am not
reporting a bug.  There's no minimum reproducible recipe, no error
to report, no surprising behaviour, etc. to speak of.  We're just
discussing Emacs development... in the emacs-devel mailing list.
I can't understand what is discourteous about this.


> I'm imagining that traversing a directory tree with an arbitrary
> predicate is going to be slow. If the predicate is limited somehow (e.g.
> to a list of "markers" as base file name, or at least wildcards), 'git
> ls-files' can probably handle this, with certain but bounded cost.
>

If the user wants marker files to mark the roots of subprojects, we'll
have to access the file system eventually because that's where
that information lives.  It would be a minimal and essential access.  If the
user is discontent with that performance hit (I really doubt it), then
the user may use other means to mark roots of subprojects, like
the one I suggested earlier.

In particular, I don't understand where "traversing a directory tree" comes
in.  That part is completely absent from the solution I am putting forth.

João

[-- Attachment #2: Type: text/html, Size: 2529 bytes --]

Re: Subprojects in project.el (Was: Eglot, project.el, and python virtual environments)

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 2197 bytes --]

On Thu, Nov 24, 2022 at 6:22 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > From: João Távora <joaotavora@gmail.com>
> > Date: Wed, 23 Nov 2022 20:33:04 +0000
> > Cc: Stefan Monnier <monnier@iro.umontreal.ca>,
> >  Danny Freeman <danny@dfreeman.email>,
> >  Eric Abrahamsen <eric@ericabrahamsen.net>, emacs-devel <
> emacs-devel@gnu.org>
> >
> >   (defun joaot/sub-projects (dir)
> >     (let* ((super (project-current))
> >            (super-root (project-root super))
> >            (relative (file-relative-name dir super-root))
> >            (d (car (split-string relative "/"))))
> >       (when (member d '("lisp" "src"))
> >         (cons 'transient (expand-file-name (concat d "/") super-root)))))
>
> Is this okay for code outside project.el to know the internal structure of
> a
> project object in general, and in particular cons "by hand" a 'transient'
> project object?  I thought this was a closely-guarded implementation detail
> that even doc strings are not allowed to divulge?
>

I don't think it's okay, no.  But I don't think project.el offers
constructors
for project objects (if it does, then I am happy to use them), so I used
this (cons 'transient...) as a placeholder and a demonstration.

Another very similar way to go about this would be for joaot/sub-projects
to be named project-sub-products and live in project.el, where it would
be free to access its own internals.

(defvar project-sub-project-prefixes nil
  "List of strings designating sub-projects.
This variable should be set directory-locally.")

(defun project-sub-projects (dir)
  (let* ((super (project-current))
         (super-root (project-root super))
         (relative (file-relative-name dir super-root))
         (d (car (split-string relative "/"))))
    (cl-loop for prefix in project-sub-project-prefixes
             when (string-prefix-p prefix relative)
             return (cons 'transient (expand-file-name prefix
super-root)))))

Then it would be a question of setting project-sub-project-prefixes
to, say, '("lisp/progmodes" "test" "doc" "src") in Emacs's .dir-locals
(or using dir-locals-set-class-variables as an alternative).

João

[-- Attachment #2: Type: text/html, Size: 3201 bytes --]

Re: Subprojects in project.el (Was: Eglot, project.el, and python virtual environments)

[Dmitry Gutov]

On 24/11/22 08:23, Eli Zaretskii wrote:
>> From: João Távora<joaotavora@gmail.com>
>> Date: Wed, 23 Nov 2022 20:33:04 +0000
>> Cc: Stefan Monnier<monnier@iro.umontreal.ca>,
>>   Danny Freeman<danny@dfreeman.email>,
>>   Eric Abrahamsen<eric@ericabrahamsen.net>, emacs-devel<emacs-devel@gnu.org>
>>
>>    (defun joaot/sub-projects (dir)
>>      (let* ((super (project-current))
>>             (super-root (project-root super))
>>             (relative (file-relative-name dir super-root))
>>             (d (car (split-string relative "/"))))
>>        (when (member d '("lisp" "src"))
>>          (cons 'transient (expand-file-name (concat d "/") super-root)))))
> Is this okay for code outside project.el to know the internal structure of a
> project object in general, and in particular cons "by hand" a 'transient'
> project object?  I thought this was a closely-guarded implementation detail
> that even doc strings are not allowed to divulge?

Joao's code here is not production-ready, but it wasn't intended as such 
either, I guess.

Any 'sub-projects' function would likely be generic with implementations 
belonging to a particular backend. And a backend knows how to construct 
its instances.

Re: Subprojects in project.el (Was: Eglot, project.el, and python virtual environments)

[Dmitry Gutov]

On 24/11/22 10:42, João Távora wrote:
> On Thu, Nov 24, 2022 at 2:49 AM Dmitry Gutov <dgutov@yandex.ru 
> <mailto:dgutov@yandex.ru>> wrote:
> 
>     On 23/11/22 22:33, João Távora wrote:
>      > For my idea to work, project-current would have to be changed to
>      >
>      > 1. Call all members of project-find-functions: return the innermost
>      >     project found thus far.
>      > 2. Not call project-find-functions recursively when called from
>     within a
>      > function
>      >     in project-find-functions, just return the innermost project
>     found
>      >     thus far.
>      >
>      > I don't see any big performance problems in this alternative.
> 
>     Might not be much of a difference in a simple configuration, but going
>     up and down dirs is not free even on a fast SSD, especially as the
>     number of the elements on this hook grows.
> 
>     But where you will definitely see a difference, is when using this with
>     Tramp.
> 
> 
> Have you seen the code I posted? You'll notice that it doesn't do
> any file system operations, it just does file name operations. And
> those run in CPU/memory, not file system. So I cannot understand
> your argument.  But I am happy to measure performance in different
> scenarios, if you'd like to suggest a particularly problematic one.

I am commenting here on the idea to "Call all members of 
project-find-functions". Members of project-find-functions do file 
system operations.

And that's what your piece of code does (call all members):

+(defvar project--overriding-projects nil)
+
  (defun project--find-in-directory (dir)
-  (run-hook-with-args-until-success 'project-find-functions dir))
+  "Find projects that directory DIR belongs to.
+Return them as a list.  If more than one is found, the first is
+usually the innermost."
+  (or project--overriding-projects
+      (let (projects)
+        (run-hook-wrapped
+         'project-find-functions
+         (lambda (fn)
+           (let* ((project--overriding-projects projects)
+                  (prj (funcall fn dir)))
+             (when prj (push prj projects)))
+           nil))
+        projects)))

Re: Subprojects in project.el (Was: Eglot, project.el, and python virtual environments)

[Tim Cross]

João Távora <joaotavora@gmail.com> writes:

> On Thu, Nov 24, 2022 at 3:01 AM Dmitry Gutov <dgutov@yandex.ru> wrote:
>
>  
>>  I'm imagining that traversing a directory tree with an arbitrary 
>>  predicate is going to be slow. If the predicate is limited somehow (e.g. 
>>  to a list of "markers" as base file name, or at least wildcards), 'git 
>>  ls-files' can probably handle this, with certain but bounded cost.

I've seen references to superior performance benefits of git ls-file a
couple of times in this thread, which has me a little confused.

There has been lots in other threads regarding the importance of not
relying on and not basing development on an underlying assumption
regarding the VCS being used. For example, I would expect project.el to
be completely neutral with respect to the VCS used in a project.

So how is git ls-file at all relevant when discussing performance
characteristics when identifying files in a project?

I also wonder if some of the performance concerns may be premature. I've
seen references to poor performance in projects with 400k or even 100k
files. What is the expected/acceptable performance for projects of that
size? How common are projects of that size? When considering
performance, are we not better off focusing on the common case rather
than extreme cases, leaving the extremes for once we have a known
problem we can then focus in on?   

Re: Subprojects in project.el (Was: Eglot, project.el, and python virtual environments)

[Dmitry Gutov]

On 24/11/22 10:50, João Távora wrote:
> On Thu, Nov 24, 2022 at 3:01 AM Dmitry Gutov <dgutov@yandex.ru 
> <mailto:dgutov@yandex.ru>> wrote:
> 
>     On 23/11/22 15:57, João Távora wrote:
>      >> It would be much more helpful in a dedicated bug report where we
>     could
>      >> discuss the details, collect the votes and see what kind of design
>      >> will ultimately satisfy the requirements. Instead of drowning it all
>      >> in this thread, which is only moderately related.
>      > I think we're doing fine here but I've changed the subject line to
>      > "unbury" it from the thread.
> 
>     You're fussy about proper bug reporting in your projects, but
>     somehow do
>     not extend that courtesy to others.
> 
> 
> I am indeed "fussy" about "bug reporting".  But here, Dmitry, I am not
> reporting a bug.  There's no minimum reproducible recipe, no error
> to report, no surprising behaviour, etc. to speak of.  We're just
> discussing Emacs development... in the emacs-devel mailing list.

You are making a new feature request. Or at least were (when you were 
talking about "subprojects" as a new noun for project.el to have, with 
new associated behaviors).

Since we're back to examining customizable ways to mark projects, I 
invite you to read 
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=41572#82 too.

> I can't understand what is discourteous about this.

That would be not following the procedure the maintainer has asked you 
to follow.

>     I'm imagining that traversing a directory tree with an arbitrary
>     predicate is going to be slow. If the predicate is limited somehow
>     (e.g.
>     to a list of "markers" as base file name, or at least wildcards), 'git
>     ls-files' can probably handle this, with certain but bounded cost.
> 
> 
> If the user wants marker files to mark the roots of subprojects, we'll
> have to access the file system eventually because that's where
> that information lives.  It would be a minimal and essential access.  If the
> user is discontent with that performance hit (I really doubt it), then
> the user may use other means to mark roots of subprojects, like
> the one I suggested earlier.
> 
> In particular, I don't understand where "traversing a directory tree" comes
> in.  That part is completely absent from the solution I am putting forth.

Your solution looks very similar to the patch in 
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=41572#85, which see.

Note that I have replied to your solution as well with a couple of 
questions, related to what is already written in the bug above.

I don't really know what "the user wants". People apparently find this 
discussion too scary or meandering to provide any additional input. The 
several who I asked to comment have walked away perplexed. Or perhaps 
it's just Debbugs.

People do seem it natural to express their custom project structures via 
file markers, at least that's what I see in the wild as 
project-find-functions customizations.

They have an upside: it's economical to customize markers once, and then 
have them used for every project of yours, instead of having to edit 
dir-locals.el everywhere where needed. But the downside is that 
implementing the non-intersecting model of project becomes expensive: in 
the most general case one will have to traverse the dir tree to find the 
inner projects to exclude them. But if the markers are plain file names 
or wildcards, optimizations are possible (e.g. 'git ls-files'; which is 
still not instant, however), at least for Git/Hg parent projects.

The idea of customizing the projects with a list of relative subproject 
directory file names solves those downsides, but comes with lack of 
automation: you have to do it for every relevant project, and not forget 
to update the settings as the project structure changes. Which might 
also be a pain e.g. when switching branches, if your dir-locals.el is 
not checked in.

Perhaps we'll ultimately end up with both ways to do this inside 
project.el, but that feels redundant.

Re: Subprojects in project.el (Was: Eglot, project.el, and python virtual environments)

[Dmitry Gutov]

On 25/11/22 00:46, Tim Cross wrote:
> 
> João Távora <joaotavora@gmail.com> writes:
> 
>> On Thu, Nov 24, 2022 at 3:01 AM Dmitry Gutov <dgutov@yandex.ru> wrote:
>>
>>   
>>>   I'm imagining that traversing a directory tree with an arbitrary
>>>   predicate is going to be slow. If the predicate is limited somehow (e.g.
>>>   to a list of "markers" as base file name, or at least wildcards), 'git
>>>   ls-files' can probably handle this, with certain but bounded cost.
> 
> I've seen references to superior performance benefits of git ls-file a
> couple of times in this thread, which has me a little confused.
> 
> There has been lots in other threads regarding the importance of not
> relying on and not basing development on an underlying assumption
> regarding the VCS being used. For example, I would expect project.el to
> be completely neutral with respect to the VCS used in a project.

That's the situation where we can optimize this case: when a project is 
Git/Hg.

> So how is git ls-file at all relevant when discussing performance
> characteristics when identifying files in a project?

Not files, though. Subprojects. Meaning, listing all (direct and 
indirect) subdirectories which satisfy a particular predicate. If the 
predicate is simple (has a particular project marker: file name or 
wildcard), it can be fetched in one shell command, like:

git ls-files -co -- "Makefile" "package.json"

(which will traverse the directory tree for you, but will also use Git's 
cache).

If the predicate is arbitrary (i.e. implemented in Lisp), the story 
would become harder.

> I also wonder if some of the performance concerns may be premature. I've
> seen references to poor performance in projects with 400k or even 100k
> files. What is the expected/acceptable performance for projects of that
> size? How common are projects of that size? When considering
> performance, are we not better off focusing on the common case rather
> than extreme cases, leaving the extremes for once we have a known
> problem we can then focus in on?

OT1H, large projects are relatively rare. OT2H, having a need for 
subprojects seems to be correlated with working on large projects.

What is the common case, in your experience, and how is it better 
solved? Globally customizing a list of "markers", or customizing a list 
of subprojects for every "parent" project?

Re: Subprojects in project.el

[Stefan Monnier]

> The idea of customizing the projects with a list of relative subproject
> directory file names solves those downsides, but comes with lack of
> automation: you have to do it for every relevant project, and not forget to
> update the settings as the project structure changes. Which might also be
> a pain e.g. when switching branches, if your dir-locals.el is not
> checked in.
>
> Perhaps we'll ultimately end up with both ways to do this inside project.el,
> but that feels redundant.

How 'bout something more "organic":
allow the user to interactively indicate "this is the root of the
current project" and save that info in some customization file?

IOW instead of having the users edit the customization info by hand,
allow them to edit it indirectly when they notice that Emacs doesn't
give the result they expected?


        Stefan

Re: Subprojects in project.el (Was: Eglot, project.el, and python virtual environments)

[Bozhidar Batsov]

[-- Attachment #1: Type: text/plain, Size: 3808 bytes --]

Here are my 2 (very generic) cents on the subject.

I'll just mention that sub-projects have been haunting me for years in Projectile, so you definitely will want to think long and hard about their implementation as people tend to have all sorts of setups. Sometimes I even wonder if it's worth it to try to support every use-case possible as it's definitely a path of growing complexity and diminishing returns. 

I definitely agree that such big and complex projects are the exception, not the norm, so I definitely wouldn't optimize for them, but rather aim to support them in the least complex and more common way. (e.g. Projectile mostly focuses on marking subprojects with `.projectile` markers and git submodules). Obviously there are many ways to approach this, but at the end of the day I'm always thinking about how common would such projects be in the real world and whether there are reasonable workarounds as an alternative to supporting them "natively/out-of-the-box". Given that project.el has been out for years and this topic comes up just now, clearly there's not great demand for subprojects functionality. (and I've had similar observations in the 11 years I've spent working on Projectile) 

On Fri, Nov 25, 2022, at 1:38 AM, Dmitry Gutov wrote:
> On 25/11/22 00:46, Tim Cross wrote:
> > 
> > João Távora <joaotavora@gmail.com> writes:
> > 
> >> On Thu, Nov 24, 2022 at 3:01 AM Dmitry Gutov <dgutov@yandex.ru> wrote:
> >>
> >>   
> >>>   I'm imagining that traversing a directory tree with an arbitrary
> >>>   predicate is going to be slow. If the predicate is limited somehow (e.g.
> >>>   to a list of "markers" as base file name, or at least wildcards), 'git
> >>>   ls-files' can probably handle this, with certain but bounded cost.
> > 
> > I've seen references to superior performance benefits of git ls-file a
> > couple of times in this thread, which has me a little confused.
> > 
> > There has been lots in other threads regarding the importance of not
> > relying on and not basing development on an underlying assumption
> > regarding the VCS being used. For example, I would expect project.el to
> > be completely neutral with respect to the VCS used in a project.
> 
> That's the situation where we can optimize this case: when a project is 
> Git/Hg.
> 
> > So how is git ls-file at all relevant when discussing performance
> > characteristics when identifying files in a project?
> 
> Not files, though. Subprojects. Meaning, listing all (direct and 
> indirect) subdirectories which satisfy a particular predicate. If the 
> predicate is simple (has a particular project marker: file name or 
> wildcard), it can be fetched in one shell command, like:
> 
> git ls-files -co -- "Makefile" "package.json"
> 
> (which will traverse the directory tree for you, but will also use Git's 
> cache).
> 
> If the predicate is arbitrary (i.e. implemented in Lisp), the story 
> would become harder.
> 
> > I also wonder if some of the performance concerns may be premature. I've
> > seen references to poor performance in projects with 400k or even 100k
> > files. What is the expected/acceptable performance for projects of that
> > size? How common are projects of that size? When considering
> > performance, are we not better off focusing on the common case rather
> > than extreme cases, leaving the extremes for once we have a known
> > problem we can then focus in on?
> 
> OT1H, large projects are relatively rare. OT2H, having a need for 
> subprojects seems to be correlated with working on large projects.
> 
> What is the common case, in your experience, and how is it better 
> solved? Globally customizing a list of "markers", or customizing a list 
> of subprojects for every "parent" project?
> 
> 

[-- Attachment #2: Type: text/html, Size: 5035 bytes --]

Re: Subprojects in project.el (Was: Eglot, project.el, and python virtual environments)

[Eli Zaretskii]

> Date: Fri, 25 Nov 2022 00:11:29 +0200
> Cc: monnier@iro.umontreal.ca, danny@dfreeman.email, eric@ericabrahamsen.net,
>  emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> > Is this okay for code outside project.el to know the internal structure of a
> > project object in general, and in particular cons "by hand" a 'transient'
> > project object?  I thought this was a closely-guarded implementation detail
> > that even doc strings are not allowed to divulge?
> 
> Joao's code here is not production-ready, but it wasn't intended as such 
> either, I guess.
> 
> Any 'sub-projects' function would likely be generic with implementations 
> belonging to a particular backend. And a backend knows how to construct 
> its instances.

I'm probably missing something: how does a Lisp program construct an
instance of a project with a known backend?  For example, if the project is
of the 'transient' kind, how would such a Lisp program go about constructing
an instance without exposing/knowing about the internals of the project
object?  I see no generic make-project API that such a Lisp program could
call.  What did I miss?

Re: Subprojects in project.el

[Juri Linkov]

> The idea of customizing the projects with a list of relative subproject
> directory file names solves those downsides, but comes with lack of
> automation: you have to do it for every relevant project, and not forget to
> update the settings as the project structure changes. Which might also be
> a pain e.g. when switching branches, if your dir-locals.el is not checked
> in.

There is no need to change dir-locals.el.  The project root can be
defined using 'dir-locals-set-class-variables'.  Then it can be
assigned to any directory using 'dir-locals-set-directory-class'.

Re: "Backend completion style" as a first-class library. Re: Eglot, project.el, and python virtual environments

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 906 bytes --]

Please have a look at this branch, and tell me what you think:

To git.sv.gnu.org:/srv/git/emacs.git
 * [new branch]            scratch/backend-completion ->
scratch/backend-completion

On Wed, Nov 23, 2022 at 1:45 AM Stefan Monnier <monnier@iro.umontreal.ca>
wrote:

> > Speaking of "ignoring scale", I've been meaning to propose here that
> > the "Backend completion style" you once wrote, Stefan, be added
> > to Emacs and to GNU Elpa as a separate :core package.
>
> Please do.  I've been meaning to do that for quite a while but it never
> reaches the top of my todo list.
>
> > So the idea is to create a `lisp/progmodes/backend-completion.lisp` with
> > the  generalized contents of what is now the ";;; Backend completion"
> > section of eglot.el.  WDYT, Stefan et al?
>
> Sounds about right, modulo the obvious s/.lisp/.el/
>
>
>         Stefan
>
>

-- 
João Távora

[-- Attachment #2: Type: text/html, Size: 1395 bytes --]

Re: Subprojects in project.el

[Stefan Monnier]

> I'll just mention that sub-projects have been haunting me for years in
> Projectile, so you definitely will want to think long and hard about their
> implementation as people tend to have all sorts of setups. Sometimes I even
> wonder if it's worth it to try to support every use-case possible as it's
> definitely a path of growing complexity and diminishing returns. 

Thanks for that background.  Personally, as a mere user of `project.el`
I wonder about the meaning of "subproject".  Are we talking about two
*separate* projects that "happen" to be layed out in the filesystem in
such a way that one is in a subdir of the other, or are we really
talking about a situation where some project operations (but not all)
will operate on "the project and its subprojects", which would then beg
the question of how to decide which operations do that (and when).


        Stefan

Re: Subprojects in project.el (Was: Eglot, project.el, and python virtual environments)

[Dmitry Gutov]

On 25/11/22 09:30, Eli Zaretskii wrote:
> I'm probably missing something: how does a Lisp program construct an
> instance of a project with a known backend?  For example, if the project is
> of the 'transient' kind, how would such a Lisp program go about constructing
> an instance without exposing/knowing about the internals of the project
> object?  I see no generic make-project API that such a Lisp program could
> call.  What did I miss?

The Lisp program that constructs instances of 'transient' is 
'project-current'. There could also someday be a new function that we 
add by its side (e.g. a helper function) which would also do that. And 
that's okay because they belong to the same package, and thus do not 
export the "private" knowledge.

The program that creates instances of 'vc' type is called 
'project-try-vc'. But we could similarly add a generic method belonging 
to the same unit of code (called 'project-subprojects') which would 
"belong" to the VC project backend and create a list of instances of its 
type.

Re: Subprojects in project.el (Was: Eglot, project.el, and python virtual environments)

[Eli Zaretskii]

> Date: Fri, 25 Nov 2022 19:24:26 +0200
> Cc: joaotavora@gmail.com, monnier@iro.umontreal.ca, danny@dfreeman.email,
>  eric@ericabrahamsen.net, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> On 25/11/22 09:30, Eli Zaretskii wrote:
> > I'm probably missing something: how does a Lisp program construct an
> > instance of a project with a known backend?  For example, if the project is
> > of the 'transient' kind, how would such a Lisp program go about constructing
> > an instance without exposing/knowing about the internals of the project
> > object?  I see no generic make-project API that such a Lisp program could
> > call.  What did I miss?
> 
> The Lisp program that constructs instances of 'transient' is 
> 'project-current'.

I meant to ask about building customized project objects, not the standard
ones that are already created by project-current.  This started with a
function João wrote to that effect, and he wrote it, AFAIU, because the
standard kinds of project objects didn't satisfy his needs.

> The program that creates instances of 'vc' type is called 
> 'project-try-vc'. But we could similarly add a generic method belonging 
> to the same unit of code (called 'project-subprojects') which would 
> "belong" to the VC project backend and create a list of instances of its 
> type.

Again, you are talking about project kinds already supported by project.el
as its built-ins.  I'm asking how to produce a project of a custom type.

Re: Subprojects in project.el

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

>> Have you seen the code I posted? You'll notice that it doesn't do
>> any file system operations, it just does file name operations. And
>> those run in CPU/memory, not file system. So I cannot understand
>> your argument.  But I am happy to measure performance in different
>> scenarios, if you'd like to suggest a particularly problematic one.
>
> I am commenting here on the idea to "Call all members of
> project-find-functions". Members of project-find-functions do file
> system operations.
>
> And that's what your piece of code does (call all members):

I don't expect a large number of member functions to exist there,
though.  At most one function looks for .git and another looks for
markers like venv (or like in my example doesn't look for markers at
all).

I really don't expect this to be a performance sink.  But since it's
easy to be wrong in these guesses, we can tweak the function that looks
for marker files .venv to only search if a current parent project is
there, just like my example.

If, on the other hand, the user's goal is to search for _both_ .git or
.venv (maybe she has some non-git venv projects) then we're just going
to have to look for two files and that's that.

João

Re: Subprojects in project.el

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

>> I am indeed "fussy" about "bug reporting".  But here, Dmitry, I am
>> not
>> reporting a bug.  There's no minimum reproducible recipe, no error
>> to report, no surprising behaviour, etc. to speak of.  We're just
>> discussing Emacs development... in the emacs-devel mailing list.
>
> You are making a new feature request. Or at least were (when you were
> talking about "subprojects" as a new noun for project.el to have, with
> new associated behaviors).

I'm discussing a limitation of project.el regarding subprojects.  If
that is solved with a new feature, a new package, or if it turns out
it's not a limitation at all, I think it's a worthy topic of
emacs-devel.

>> I can't understand what is discourteous about this.
> That would be not following the procedure the maintainer has asked you
> to follow.

If that means silencing me on emacs-devel, then you're out of luck.

> I don't really know what "the user wants". People apparently find this
> discussion too scary or meandering to provide any additional
> input. The several who I asked to comment have walked away
> perplexed. Or perhaps it's just Debbugs.

People seem to be contributin a healthy amount of information here.

> People do seem it natural to express their custom project structures
> via file markers, at least that's what I see in the wild as
> project-find-functions customizations.

Yes.  Very often there are already such markers in place.  Other times
you can add them yourself.  Other time there aren't any and the people
controlling the projects don't want you to add them (maybe because they
don't use Emacs or care about your uses).

But that shouldn't matter.

My understanding of subprojects, or the operations I want to do with
them isn't affected by the method by which they may be configured:
that's a detail that relatively easy to solve.

To be clear, here's my use case again: I have a complex hierarchy of
directories and files which call the super-project.  I sometimes want to
find files, grep for strings and run compile commands there.  project.el
allows this already (albeit with associated find-file slowness if the
project is really large).

Sometimes I will work for some period exclusively in one of the
super-projects's sub hierarchies.  When doing so, I will look for files,
grep strings and run compile commands in that hierarchy which I call the
sub-project.  Doing so cuts down on the noise of other files and grep
matches in other parts of the super-project that I'm not interested in.

Not all files that belong to the super-project necessarily belong to a
sub-project.  Some of them _only_ belong to the super-project.n

Anyway, indeally I want these three main operations (find-file, grep,
compile) to run in the inner sub-project by default.  By typing
something more, like, say, a negative prefix argument, I want to be able
to be given the possibility to operate on the super-project instead.

> The idea of customizing the projects with a list of relative
> subproject directory file names solves those downsides, but comes with
> lack of automation: you have to do it for every relevant project, and
> not forget to update the settings as the project structure
> changes. Which might also be a pain e.g. when switching branches, if
> your dir-locals.el is not checked in.

As Juri mentioned, dir-locals-set-class-variables is the tool you need
in those cases.  There are ample tools to solve this problem.  We should
first focus on the project.el infrastructure that enables the above use
case in the first place.

João

Re: Subprojects in project.el

[João Távora]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> The idea of customizing the projects with a list of relative subproject
>> directory file names solves those downsides, but comes with lack of
>> automation: you have to do it for every relevant project, and not forget to
>> update the settings as the project structure changes. Which might also be
>> a pain e.g. when switching branches, if your dir-locals.el is not
>> checked in.
>>
>> Perhaps we'll ultimately end up with both ways to do this inside project.el,
>> but that feels redundant.
>
> How 'bout something more "organic":
> allow the user to interactively indicate "this is the root of the
> current project" and save that info in some customization file?
>
> IOW instead of having the users edit the customization info by hand,
> allow them to edit it indirectly when they notice that Emacs doesn't
> give the result they expected?

I'm usually not a fan of these solutions.  If marker files are
available, they negate their benefits (i.e. it breaks when you move the
project around).

If marker files are not available then I would have some piece of
configuration lying around in some special place, format and language
that is not the usual elisp.  The same configuration might as well have
been coded with existing idioms (i.e. dir-locals-set-directory-class)
and kept as Lisp in my init file.

That said, I'm not against adding yet another project-definition
facility that project.el can be made to understand.  It's not hard to
imagine a member of project-find-functions supporting the customization
file you suggest.

João

Re: Subprojects in project.el

[Dmitry Gutov]

On 25/11/22 09:42, Juri Linkov wrote:
>> The idea of customizing the projects with a list of relative subproject
>> directory file names solves those downsides, but comes with lack of
>> automation: you have to do it for every relevant project, and not forget to
>> update the settings as the project structure changes. Which might also be
>> a pain e.g. when switching branches, if your dir-locals.el is not checked
>> in.
> There is no need to change dir-locals.el.  The project root can be
> defined using 'dir-locals-set-class-variables'.  Then it can be
> assigned to any directory using 'dir-locals-set-directory-class'.

How would we fetch the exact root at runtime, if the value is applied to 
the whole directory subtree?

Re: Subprojects in project.el

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

>> I also wonder if some of the performance concerns may be premature. I've
>> seen references to poor performance in projects with 400k or even 100k
>> files. What is the expected/acceptable performance for projects of that
>> size? How common are projects of that size? When considering
>> performance, are we not better off focusing on the common case rather
>> than extreme cases, leaving the extremes for once we have a known
>> problem we can then focus in on?
>
> OT1H, large projects are relatively rare. OT2H, having a need for
> subprojects seems to be correlated with working on large projects.

There are medium-sized projects where sub-projects makes a lot of sense
and the performance of C-x p f project-find-file is fine for both the
super-project and the sub-project.

But it's true that the larger the project, the more likely the need for
subprojects.  And it's also true, as you note, that sub-projects C-x p f
might surprisingly be less performant than in the super-project.

But IMO optinion the slowness of C-x p f in large projects isn't solved
by git ls-files in my opinion.  That can only do so much.  Once the
project grows large enough in files, git ls-files can be quick as
lighting, but you just won't be able to cons that ginormous list of
files in the project-files implementation.  This is why it's important
to allow project-files to return a generalized completion table.

To write those tables, you will need an external tool (like "voidtool
everything" on MS Windows, or GNU locate on Linux) and the "backend
completion" style that I'm implementing with Stefan.

João

Re: Subprojects in project.el (Was: Eglot, project.el, and python virtual environments)

[Dmitry Gutov]

On 25/11/22 21:45, Eli Zaretskii wrote:
>> Date: Fri, 25 Nov 2022 19:24:26 +0200
>> Cc: joaotavora@gmail.com, monnier@iro.umontreal.ca, danny@dfreeman.email,
>>   eric@ericabrahamsen.net, emacs-devel@gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>> On 25/11/22 09:30, Eli Zaretskii wrote:
>>> I'm probably missing something: how does a Lisp program construct an
>>> instance of a project with a known backend?  For example, if the project is
>>> of the 'transient' kind, how would such a Lisp program go about constructing
>>> an instance without exposing/knowing about the internals of the project
>>> object?  I see no generic make-project API that such a Lisp program could
>>> call.  What did I miss?
>>
>> The Lisp program that constructs instances of 'transient' is
>> 'project-current'.
> 
> I meant to ask about building customized project objects, not the standard
> ones that are already created by project-current.  This started with a
> function João wrote to that effect, and he wrote it, AFAIU, because the
> standard kinds of project objects didn't satisfy his needs.

No, again, he just wanted to show a quick example. With pseudocode, 
almost like.

>> The program that creates instances of 'vc' type is called
>> 'project-try-vc'. But we could similarly add a generic method belonging
>> to the same unit of code (called 'project-subprojects') which would
>> "belong" to the VC project backend and create a list of instances of its
>> type.
> 
> Again, you are talking about project kinds already supported by project.el
> as its built-ins.  I'm asking how to produce a project of a custom type.

To produce a project of custom type, you decide the data structure for 
that type, write implementations for all the required (and perhaps some 
optional) generic methods, then create that structure.

You asked, however, how to instantiate a project of a type belonging to 
"someone else". But didn't explain why.

In most cases we will say it's not a good idea, but when a practical 
goal is described we will decide to either say "go ahead, it's okay in 
this case", or, hopefully, suggest a different way to reach that goal 
(just like I did with the 'project-parent' definition example). Or 
rethink and throw away the whole design (hopefully not).

The reasons not to rely on internal structure, as a reminder:

- The internal structures are prone to change, and you don't want your 
code to break when that happens.
- When implementing some feature that works with projects, you generally 
want it to work with all kinds of projects (which was the whole point of 
project.el -- to make this approach work). And that would mean talking 
to them through common methods, rather that examine the internals of 
this or that project type.

Re: Subprojects in project.el

[Dmitry Gutov]

On 25/11/22 04:38, Stefan Monnier wrote:
>> The idea of customizing the projects with a list of relative subproject
>> directory file names solves those downsides, but comes with lack of
>> automation: you have to do it for every relevant project, and not forget to
>> update the settings as the project structure changes. Which might also be
>> a pain e.g. when switching branches, if your dir-locals.el is not
>> checked in.
>>
>> Perhaps we'll ultimately end up with both ways to do this inside project.el,
>> but that feels redundant.
> How 'bout something more "organic":
> allow the user to interactively indicate "this is the root of the
> current project" and save that info in some customization file?
> 
> IOW instead of having the users edit the customization info by hand,
> allow them to edit it indirectly when they notice that Emacs doesn't
> give the result they expected?

I don't personally like the idea of a centralized customized list of 
project roots, though I remember some project implementations back in 
the day took that approach. Not the most popular ones, though, IIRC.

Setting stuff up via dir-locals is more of my cup of tea, to be honest 
(when manual configuration is unavoidable). But we might add your way as 
well, perhaps later.

Re: Subprojects in project.el

[Dmitry Gutov]

On 25/11/22 16:58, Stefan Monnier wrote:
>> I'll just mention that sub-projects have been haunting me for years in
>> Projectile, so you definitely will want to think long and hard about their
>> implementation as people tend to have all sorts of setups. Sometimes I even
>> wonder if it's worth it to try to support every use-case possible as it's
>> definitely a path of growing complexity and diminishing returns.
> Thanks for that background.  Personally, as a mere user of `project.el`
> I wonder about the meaning of "subproject".  Are we talking about two
> *separate*  projects that "happen" to be layed out in the filesystem in
> such a way that one is in a subdir of the other, or are we really
> talking about a situation where some project operations (but not all)
> will operate on "the project and its subprojects", which would then beg
> the question of how to decide which operations do that (and when).

Just to clarify how we got there: I only used the term "subprojects" to 
refer to the former, and only for the purpose of describing how one 
could implement a particular behavior: excluding subprojects' files from 
what 'project-files' returns for the parent project.

IOW, being able to make sure the projects are "disjoint", even if one is 
"inside" the other. And to get a performance bump from doing that, in 
certain cases.

And when the discussion got to the latter at one time (a more complex 
and open-ended topic), I asked to split that into a separate feature 
request via 'M-x report-emacs-bug'.

Re: Subprojects in project.el

[Dmitry Gutov]

On 25/11/22 21:56, João Távora wrote:
> Dmitry Gutov <dgutov@yandex.ru> writes:
> 
>>> Have you seen the code I posted? You'll notice that it doesn't do
>>> any file system operations, it just does file name operations. And
>>> those run in CPU/memory, not file system. So I cannot understand
>>> your argument.  But I am happy to measure performance in different
>>> scenarios, if you'd like to suggest a particularly problematic one.
>>
>> I am commenting here on the idea to "Call all members of
>> project-find-functions". Members of project-find-functions do file
>> system operations.
>>
>> And that's what your piece of code does (call all members):
> 
> I don't expect a large number of member functions to exist there,
> though.  At most one function looks for .git and another looks for
> markers like venv (or like in my example doesn't look for markers at
> all).

Even when there are just 2 of them, it's 2x the initial delay, which can 
already be noticeable with Tramp. E.g. my ping to the "mainland" is 
around 100ms. And filesystem operations might take several roundtrips.

Further, the approach you mentioned (running all functions on the hook) 
implied that there will be more of them than one -- for different 
project kinds, simple markers or etc. Otherwise there would be no 
difference between doing that and just what we are doing now.

> I really don't expect this to be a performance sink.  But since it's
> easy to be wrong in these guesses, we can tweak the function that looks
> for marker files .venv to only search if a current parent project is
> there, just like my example.
> 
> If, on the other hand, the user's goal is to search for _both_ .git or
> .venv (maybe she has some non-git venv projects) then we're just going
> to have to look for two files and that's that.

I don't entirely understand the algorithm you're proposing here, but it 
sounds vaguely like what I'm writing now, so you might like the result.

Re: Subprojects in project.el

[Dmitry Gutov]

On 25/11/22 22:16, João Távora wrote:
> Dmitry Gutov <dgutov@yandex.ru> writes:
> 
>>> I am indeed "fussy" about "bug reporting".  But here, Dmitry, I am
>>> not
>>> reporting a bug.  There's no minimum reproducible recipe, no error
>>> to report, no surprising behaviour, etc. to speak of.  We're just
>>> discussing Emacs development... in the emacs-devel mailing list.
>>
>> You are making a new feature request. Or at least were (when you were
>> talking about "subprojects" as a new noun for project.el to have, with
>> new associated behaviors).
> 
> I'm discussing a limitation of project.el regarding subprojects.  If
> that is solved with a new feature, a new package, or if it turns out
> it's not a limitation at all, I think it's a worthy topic of
> emacs-devel.

Like Stefan also explained, there are two different things one might be 
talking about when talking about "subprojects".

>>> I can't understand what is discourteous about this.
>> That would be not following the procedure the maintainer has asked you
>> to follow.
> 
> If that means silencing me on emacs-devel, then you're out of luck.

Is that what you do when you ask somebody to use the bug tracker?

>> I don't really know what "the user wants". People apparently find this
>> discussion too scary or meandering to provide any additional
>> input. The several who I asked to comment have walked away
>> perplexed. Or perhaps it's just Debbugs.
> 
> People seem to be contributin a healthy amount of information here.

Yes and no. Nobody has bothered to comment on the messages in the bug 
report, or on the patches in it.

>> People do seem it natural to express their custom project structures
>> via file markers, at least that's what I see in the wild as
>> project-find-functions customizations.
> 
> Yes.  Very often there are already such markers in place.  Other times
> you can add them yourself.  Other time there aren't any and the people
> controlling the projects don't want you to add them (maybe because they
> don't use Emacs or care about your uses).
> 
> But that shouldn't matter.
> 
> My understanding of subprojects, or the operations I want to do with
> them isn't affected by the method by which they may be configured:
> that's a detail that relatively easy to solve.

Have you read the bug I linked to? You don't need to explain something I 
already know.

The technical solutions for this are plenty. The question is how to make 
a coherent solution from that, and to address the most common scenarios.

> To be clear, here's my use case again: I have a complex hierarchy of
> directories and files which call the super-project.  I sometimes want to
> find files, grep for strings and run compile commands there.  project.el
> allows this already (albeit with associated find-file slowness if the
> project is really large).
> 
> Sometimes I will work for some period exclusively in one of the
> super-projects's sub hierarchies.  When doing so, I will look for files,
> grep strings and run compile commands in that hierarchy which I call the
> sub-project.  Doing so cuts down on the noise of other files and grep
> matches in other parts of the super-project that I'm not interested in.

Note that it would also be possible to do through some other means. E.g. 
using some command in Xref result buffer which would filter by file 
names and hide the rest.

> Not all files that belong to the super-project necessarily belong to a
> sub-project.  Some of them _only_ belong to the super-project.n

Do the files that belong to a sub-project belong to the super-project?

> Anyway, indeally I want these three main operations (find-file, grep,
> compile) to run in the inner sub-project by default.  By typing
> something more, like, say, a negative prefix argument, I want to be able
> to be given the possibility to operate on the super-project instead.

See the 'project-parent' implementation I posted a couple of days ago.

>> The idea of customizing the projects with a list of relative
>> subproject directory file names solves those downsides, but comes with
>> lack of automation: you have to do it for every relevant project, and
>> not forget to update the settings as the project structure
>> changes. Which might also be a pain e.g. when switching branches, if
>> your dir-locals.el is not checked in.
> 
> As Juri mentioned, dir-locals-set-class-variables is the tool you need
> in those cases.  There are ample tools to solve this problem.  We should
> first focus on the project.el infrastructure that enables the above use
> case in the first place.

What's missing in the infrastructure?

Re: Subprojects in project.el

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 25/11/22 09:42, Juri Linkov wrote:
>>> The idea of customizing the projects with a list of relative subproject
>>> directory file names solves those downsides, but comes with lack of
>>> automation: you have to do it for every relevant project, and not forget to
>>> update the settings as the project structure changes. Which might also be
>>> a pain e.g. when switching branches, if your dir-locals.el is not checked
>>> in.
>> There is no need to change dir-locals.el.  The project root can be
>> defined using 'dir-locals-set-class-variables'.  Then it can be
>> assigned to any directory using 'dir-locals-set-directory-class'.
>
> How would we fetch the exact root at runtime, if the value is applied
> to the whole directory subtree?

You can use dir-locals-set-class-variables to variable like the
hypothetical project-subproject-prefixes to the super-project's root.
This contains a list of strings or regexps that identify the subprojects
inside the superproject.  I don't see the problem.  Elements in
project-find-functions would be aware of this value and proceed
accordingly to find the subprojects.

João

Re: Subprojects in project.el

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

> No, again, he just wanted to show a quick example. With pseudocode,
> almost like.

It was real, working code.  The only thing that was missing was uniform
way to take advantage of the new structure in the user commands to find
files, grep strings and run compilation commands.  Also, the follow-up
message showed that it's possible to have the new logic and not use any
internal data structures "from the outside".

>>> The program that creates instances of 'vc' type is called
>>> 'project-try-vc'. But we could similarly add a generic method belonging
>>> to the same unit of code (called 'project-subprojects') which would
>>> "belong" to the VC project backend and create a list of instances of its
>>> type.
>> Again, you are talking about project kinds already supported by
>> project.el
>> as its built-ins.  I'm asking how to produce a project of a custom type.
>
> To produce a project of custom type, you decide the data structure for
> that type, write implementations for all the required (and perhaps
> some optional) generic methods, then create that structure.
>
> You asked, however, how to instantiate a project of a type belonging
> to "someone else". But didn't explain why.
>
> In most cases we will say it's not a good idea, but when a practical
> goal is described we will decide to either say "go ahead, it's okay in
> this case", or, hopefully, suggest a different way to reach that goal
> (just like I did with the 'project-parent' definition example). Or
> rethink and throw away the whole design (hopefully not).

I don't think it has to be so extreme.  I don't understand why there
isn't a user-callable construtor for a type of project that is currently
represented by the '(transient . "<dir>") implementation detail.
Demanding that the users of looking to add to project-find-functions
additionally define a whole new type and all the operations to go with
it is unreasonable, IMO.  I don't think it's "throwing away the whole
design" to provide one such constructor or a means to simplify this.

Even better, provide a CLOS class, so that users may subclass it and use
inheritance in the CLOS generics.

João

Re: Subprojects in project.el

[Dmitry Gutov]

On 26/11/22 01:47, João Távora wrote:
> You can use dir-locals-set-class-variables to variable like the
> hypothetical project-subproject-prefixes to the super-project's root.
> This contains a list of strings or regexps that identify the subprojects
> inside the superproject.  I don't see the problem.  Elements in
> project-find-functions would be aware of this value and proceed
> accordingly to find the subprojects.

I don't see the problem in that.

I just fail to see an advantage either, given that the list of prefixes 
would be different between the "super" projects, so the suggestion to 
use a class is perplexing.

Re: Subprojects in project.el

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

> Even when there are just 2 of them, it's 2x the initial delay, which
> can already be noticeable with Tramp. E.g. my ping to the "mainland"
> is around 100ms. And filesystem operations might take several
> roundtrips.

If you want to use subprojects over TRAMP and the super-project and the
sub-project are both identified by marker files files, then it's not
something we can avoid.  IOW we do just as many marker-file probes as
needed: you pay for what you get.  If the user wants out of the bargain,
he can customize one of the search functions to give up in TRAMP
contexts.

> I don't entirely understand the algorithm you're proposing here, but
> it sounds vaguely like what I'm writing now, so you might like the
> result.

Where/when can I find this code?

Re: Subprojects in project.el

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

>>>> I can't understand what is discourteous about this.
>>> That would be not following the procedure the maintainer has asked you
>>> to follow.
>> If that means silencing me on emacs-devel, then you're out of luck.
>
> Is that what you do when you ask somebody to use the bug tracker?

I'll use the bug tracker when I think it's appropriate.  Let's not
insinuate I'm some kind of inconsiderate delinquent for not moving the
discussion there as you would want.  I'm not reporting a bug and I've
politely declined your suggestion, so stop beating this horse.

>>> I don't really know what "the user wants". People apparently find this
>>> discussion too scary or meandering to provide any additional
>>> input. The several who I asked to comment have walked away
>>> perplexed. Or perhaps it's just Debbugs.
>> People seem to be contributin a healthy amount of information here.
>
> Yes and no. Nobody has bothered to comment on the messages in the bug
> report, or on the patches in it.

If they help the discussion, suggest you put your patches in a scratch
branch and announce it here.  It's not much work for you, I suppose and
personally I find it simpler than finding your patches and applying them
to the right version.

>>> People do seem it natural to express their custom project structures
>>> via file markers, at least that's what I see in the wild as
>>> project-find-functions customizations.
>> Yes.  Very often there are already such markers in place.  Other
>> times
>> you can add them yourself.  Other time there aren't any and the people
>> controlling the projects don't want you to add them (maybe because they
>> don't use Emacs or care about your uses).
>> But that shouldn't matter.
>> My understanding of subprojects, or the operations I want to do with
>> them isn't affected by the method by which they may be configured:
>> that's a detail that relatively easy to solve.
>
> Have you read the bug I linked to? You don't need to explain something
> I already know.

If you're already perfectly aware of it, good for you.  But this is a
public forum, I'm explaining to all the recipients of this message what
my understanding and use case is.

> Note that it would also be possible to do through some other
> means. E.g. using some command in Xref result buffer which would
> filter by file names and hide the rest.

*compilation* and *shell-command-output* are not in any kind of
structured Xref mode where that hypothetical command would operate.
Even if such a command could be devised, I don't find that don't find it
a very good solution.  If one knows the where to look in, it's better to
grep only one haystack instead of the whole barn just to throw away most
of the needles.

>> Not all files that belong to the super-project necessarily belong to a
>> sub-project.  Some of them _only_ belong to the super-project.n
> Do the files that belong to a sub-project belong to the super-project?

Yes: in my view they belong to both.  i.e. if you project-find-file in
the super-project, you can find a file in a subproject such that a
subsequent project command operates on the subproject.

>> Anyway, indeally I want these three main operations (find-file, grep,
>> compile) to run in the inner sub-project by default.  By typing
>> something more, like, say, a negative prefix argument, I want to be able
>> to be given the possibility to operate on the super-project instead.
>
> See the 'project-parent' implementation I posted a couple of days ago.

I've seen it.  In fact, I posted the same code earlier.  But how do I
plug in that so that M-- C-x p f, M-- C-x p c, etc, etc make use of it?

>>> The idea of customizing the projects with a list of relative
>>> subproject directory file names solves those downsides, but comes with
>>> lack of automation: you have to do it for every relevant project, and
>>> not forget to update the settings as the project structure
>>> changes. Which might also be a pain e.g. when switching branches, if
>>> your dir-locals.el is not checked in.
>> As Juri mentioned, dir-locals-set-class-variables is the tool you
>> need
>> in those cases.  There are ample tools to solve this problem.  We should
>> first focus on the project.el infrastructure that enables the above use
>> case in the first place.
>
> What's missing in the infrastructure?

Not much, I would say.  But I think at least:

* A way that I can add an element to project-find-functions that
  understands that a super-project has been detected already in the
  current search and proceeds to find sub-projects inside it.  This is
  what I posted code for.

* A way for M-- (or similar) to consistently affect all (or most) of the
  operations in the C-x p keymap so that we can choose if the operation
  operates on the super-project, if it exists.  Unfortunately some of
  these commands (like project-find-files) already take a prefix
  argument to mean different things.  But it's not too bad.

* A new project type, similar to the '(transient . "dir") project (and
  inheriting most of its operations) that also keeps a record of the
  super-project found.  This might not be strictly necessary, but could
  come in handy later for efficiency reasons.

João

Re: Subprojects in project.el

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 26/11/22 01:47, João Távora wrote:
>> You can use dir-locals-set-class-variables to variable like the
>> hypothetical project-subproject-prefixes to the super-project's root.
>> This contains a list of strings or regexps that identify the subprojects
>> inside the superproject.  I don't see the problem.  Elements in
>> project-find-functions would be aware of this value and proceed
>> accordingly to find the subprojects.
>
> I don't see the problem in that.
>
> I just fail to see an advantage either, given that the list of
> prefixes would be different between the "super" projects, so the
> suggestion to use a class is perplexing.

The point of dir-locals-set-class-variables is to set directly-locals
"from a distance" when creating an actual .dir-locals.el file isn't
feasible or desired.

  (dir-locals-set-directory-class "~/Source/very-big-project" 'very-big-project)
  (dir-locals-set-class-variables 'very-big-project ((nil . (project-find-prefixes "foo" "bar/baz.*"))))

Even if it's a one-use class, there's nothing wrong with that IMO. And
chances are you have multiple clones or Git worktrees of the same
project, so you may as well make it a two- or three-use class

  (dir-locals-set-directory-class "~/Source/worktree-2" 'very-big-project)
  (dir-locals-set-directory-class "~/Source/another-clone" 'very-big-project)


Of course if the user is able to AND wants to use .dir-locals.el or
marker files, that is fine, too.  I personally like the above scheme.

João

Re: Subprojects in project.el

[Dmitry Gutov]

On 26/11/22 02:00, João Távora wrote:
> Dmitry Gutov<dgutov@yandex.ru>  writes:
> 
>> Even when there are just 2 of them, it's 2x the initial delay, which
>> can already be noticeable with Tramp. E.g. my ping to the "mainland"
>> is around 100ms. And filesystem operations might take several
>> roundtrips.
> If you want to use subprojects over TRAMP and the super-project and the
> sub-project are both identified by marker files files, then it's not
> something we can avoid.  IOW we do just as many marker-file probes as
> needed: you pay for what you get.  If the user wants out of the bargain,
> he can customize one of the search functions to give up in TRAMP
> contexts.

We aren't even close to top efficiency WRT file operations now. And your 
idea is also based on hand-crafting project-find-functions to only 
contain the necessary items.

As opposed to real-life scenario of Projectile just adding its function 
at the front.

>> I don't entirely understand the algorithm you're proposing here, but
>> it sounds vaguely like what I'm writing now, so you might like the
>> result.
> Where/when can I find this code?

You and others can now find it here: https://debbugs.gnu.org/41572#186

Re: Subprojects in project.el

[Dmitry Gutov]

On 26/11/22 02:00, João Távora wrote:
>> I don't entirely understand the algorithm you're proposing here, but
>> it sounds vaguely like what I'm writing now, so you might like the
>> result.
> Where/when can I find this code?

In the bug report I've linked to a few times here. I've Cc'd you.

Re: Subprojects in project.el

[Dmitry Gutov]

On 26/11/22 02:37, João Távora wrote:
> Dmitry Gutov <dgutov@yandex.ru> writes:
> 
>>>>> I can't understand what is discourteous about this.
>>>> That would be not following the procedure the maintainer has asked you
>>>> to follow.
>>> If that means silencing me on emacs-devel, then you're out of luck.
>>
>> Is that what you do when you ask somebody to use the bug tracker?
> 
> I'll use the bug tracker when I think it's appropriate.  Let's not
> insinuate I'm some kind of inconsiderate delinquent for not moving the
> discussion there as you would want.  I'm not reporting a bug and I've
> politely declined your suggestion, so stop beating this horse.

Must be nice to be the person who gets to decide what is appropriate in 
any situation.

>>>> I don't really know what "the user wants". People apparently find this
>>>> discussion too scary or meandering to provide any additional
>>>> input. The several who I asked to comment have walked away
>>>> perplexed. Or perhaps it's just Debbugs.
>>> People seem to be contributin a healthy amount of information here.
>>
>> Yes and no. Nobody has bothered to comment on the messages in the bug
>> report, or on the patches in it.
> 
> If they help the discussion, suggest you put your patches in a scratch
> branch and announce it here.  It's not much work for you, I suppose and
> personally I find it simpler than finding your patches and applying them
> to the right version.

I suggest people look at the patches where they are posted.

>> Note that it would also be possible to do through some other
>> means. E.g. using some command in Xref result buffer which would
>> filter by file names and hide the rest.
> 
> *compilation* and *shell-command-output* are not in any kind of
> structured Xref mode where that hypothetical command would operate.
> Even if such a command could be devised, I don't find that don't find it
> a very good solution.  If one knows the where to look in, it's better to
> grep only one haystack instead of the whole barn just to throw away most
> of the needles.

Sure.

>>> Not all files that belong to the super-project necessarily belong to a
>>> sub-project.  Some of them _only_ belong to the super-project.n
>> Do the files that belong to a sub-project belong to the super-project?
> 
> Yes: in my view they belong to both.  i.e. if you project-find-file in
> the super-project, you can find a file in a subproject such that a
> subsequent project command operates on the subproject.

Okay, good.

>>> Anyway, indeally I want these three main operations (find-file, grep,
>>> compile) to run in the inner sub-project by default.  By typing
>>> something more, like, say, a negative prefix argument, I want to be able
>>> to be given the possibility to operate on the super-project instead.
>>
>> See the 'project-parent' implementation I posted a couple of days ago.
> 
> I've seen it.  In fact, I posted the same code earlier.  But how do I
> plug in that so that M-- C-x p f, M-- C-x p c, etc, etc make use of it?

By modifying each and every command. I don't think it would be 
appropriate for 'project-current' itself to react to the value of 
current-prefix-arg.

As you are aware, some commands already react to prefix argument, and 
some third-party ones might even handle the negative value. Let every 
command deal with UI; we can make a helper that will make the code much 
shorter anyway.

>>>> The idea of customizing the projects with a list of relative
>>>> subproject directory file names solves those downsides, but comes with
>>>> lack of automation: you have to do it for every relevant project, and
>>>> not forget to update the settings as the project structure
>>>> changes. Which might also be a pain e.g. when switching branches, if
>>>> your dir-locals.el is not checked in.
>>> As Juri mentioned, dir-locals-set-class-variables is the tool you
>>> need
>>> in those cases.  There are ample tools to solve this problem.  We should
>>> first focus on the project.el infrastructure that enables the above use
>>> case in the first place.
>>
>> What's missing in the infrastructure?
> 
> Not much, I would say.  But I think at least:
> 
> * A way that I can add an element to project-find-functions that
>    understands that a super-project has been detected already in the
>    current search and proceeds to find sub-projects inside it.  This is
>    what I posted code for.
> 
> * A way for M-- (or similar) to consistently affect all (or most) of the
>    operations in the C-x p keymap so that we can choose if the operation
>    operates on the super-project, if it exists.  Unfortunately some of
>    these commands (like project-find-files) already take a prefix
>    argument to mean different things.  But it's not too bad.
> 
> * A new project type, similar to the '(transient . "dir") project (and
>    inheriting most of its operations) that also keeps a record of the
>    super-project found.  This might not be strictly necessary, but could
>    come in handy later for efficiency reasons.

Sounds pretty complicated. See if the latest patch solves your immediate 
problems just as well.

Aside from the 'M--' thing, which is a compatible but separate story.

Re: Subprojects in project.el

[Dmitry Gutov]

On 26/11/22 02:46, João Távora wrote:
> Dmitry Gutov<dgutov@yandex.ru>  writes:
> 
>> On 26/11/22 01:47, João Távora wrote:
>>> You can use dir-locals-set-class-variables to variable like the
>>> hypothetical project-subproject-prefixes to the super-project's root.
>>> This contains a list of strings or regexps that identify the subprojects
>>> inside the superproject.  I don't see the problem.  Elements in
>>> project-find-functions would be aware of this value and proceed
>>> accordingly to find the subprojects.
>> I don't see the problem in that.
>>
>> I just fail to see an advantage either, given that the list of
>> prefixes would be different between the "super" projects, so the
>> suggestion to use a class is perplexing.
> The point of dir-locals-set-class-variables is to set directly-locals
> "from a distance" when creating an actual .dir-locals.el file isn't
> feasible or desired.
> 
>    (dir-locals-set-directory-class "~/Source/very-big-project" 'very-big-project)
>    (dir-locals-set-class-variables 'very-big-project ((nil . (project-find-prefixes "foo" "bar/baz.*"))))
> 
> Even if it's a one-use class, there's nothing wrong with that IMO. And
> chances are you have multiple clones or Git worktrees of the same
> project, so you may as well make it a two- or three-use class
> 
>    (dir-locals-set-directory-class "~/Source/worktree-2" 'very-big-project)
>    (dir-locals-set-directory-class "~/Source/another-clone" 'very-big-project)
> 
> 
> Of course if the user is able to AND wants to use .dir-locals.el or
> marker files, that is fine, too.  I personally like the above scheme.

We don't have the -prefixes variable (and Stefan asked for a way to tag 
a project root, probably even without any VC repository around), but as 
a replacement for .dir-locals.el -- cool, good recommendation.

Re: Subprojects in project.el

[Stefan Monnier]

>>> I'll just mention that sub-projects have been haunting me for years in
>>> Projectile, so you definitely will want to think long and hard about their
>>> implementation as people tend to have all sorts of setups. Sometimes I even
>>> wonder if it's worth it to try to support every use-case possible as it's
>>> definitely a path of growing complexity and diminishing returns.
>> Thanks for that background.  Personally, as a mere user of `project.el`
>> I wonder about the meaning of "subproject".  Are we talking about two
>> *separate*  projects that "happen" to be layed out in the filesystem in
>> such a way that one is in a subdir of the other, or are we really
>> talking about a situation where some project operations (but not all)
>> will operate on "the project and its subprojects", which would then beg
>> the question of how to decide which operations do that (and when).
>
> Just to clarify how we got there: I only used the term "subprojects" to
> refer to the former, and only for the purpose of describing how one could
> implement a particular behavior: excluding subprojects' files from what
> 'project-files' returns for the parent project.
>
> IOW, being able to make sure the projects are "disjoint", even if one is
> "inside" the other. And to get a performance bump from doing that, in
> certain cases.
>
> And when the discussion got to the latter at one time (a more complex and
>  open-ended topic), I asked to split that into a separate feature request
> via 'M-x report-emacs-bug'.

I get the impression that the discussion is a it muddled by the lack of
distinction between the two, then (at least it wasn't clear to me which
part applied to which case).

I suggest we choose two different names for those two different cases.


        Stefan

Re: Subprojects in project.el (Was: Eglot, project.el, and python virtual environments)

[Eli Zaretskii]

> Date: Fri, 25 Nov 2022 23:57:51 +0200
> Cc: joaotavora@gmail.com, monnier@iro.umontreal.ca, danny@dfreeman.email,
>  eric@ericabrahamsen.net, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> >> The Lisp program that constructs instances of 'transient' is
> >> 'project-current'.
> > 
> > I meant to ask about building customized project objects, not the standard
> > ones that are already created by project-current.  This started with a
> > function João wrote to that effect, and he wrote it, AFAIU, because the
> > standard kinds of project objects didn't satisfy his needs.
> [...]
> > Again, you are talking about project kinds already supported by project.el
> > as its built-ins.  I'm asking how to produce a project of a custom type.
> 
> To produce a project of custom type, you decide the data structure for 
> that type, write implementations for all the required (and perhaps some 
> optional) generic methods, then create that structure.

It seems I need to spell out everything every time I'm writing something
because otherwise you interpret it out of context.  So here: I'm asking how
to produce a customized project whose type is 'transient', but which doesn't
go by the rules of the built-in 'transient' project.

As for "project of custom type", where "custom" means "not one of the known
types", there's still a situation where my project is similar enough to one
of the built-in types to make reimplementation of all the APIs overkill.
One could want to use some of the APIs without change and customize others.
So this, too, is something that project.el should IMO allow without too much
fuss.

> You asked, however, how to instantiate a project of a type belonging to 
> "someone else". But didn't explain why.

The "why" is "because one needs to".

> The reasons not to rely on internal structure, as a reminder:
> 
> - The internal structures are prone to change, and you don't want your 
> code to break when that happens.
> - When implementing some feature that works with projects, you generally 
> want it to work with all kinds of projects (which was the whole point of 
> project.el -- to make this approach work). And that would mean talking 
> to them through common methods, rather that examine the internals of 
> this or that project type.

I'm fine with all that, but it sounds like the above makes it impossible to
implement custom project objects, maybe because there's no make-project
method that one could use and/or subclass.

Re: Subprojects in project.el

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 26/11/22 02:00, João Távora wrote:

> We aren't even close to top efficiency WRT file operations now. And
> your idea is also based on hand-crafting project-find-functions to
> only contain the necessary items.

project-find-functions is part of project.el's API.  So it makes a lot
of sense to use, otherwise why would it be there?  We can provide some
pre-baked functions to put there, or already put there for users.  The
key point is that functions bail out early if they don't the adequate
context.  One of them is the one I proposed earlier, which uses a local
project-subproject-prefixes variable.  Another would search for marker
files and consult a tramp-specific user variable for opt-out.

> As opposed to real-life scenario of Projectile just adding its
> function at the front.
>
>>> I don't entirely understand the algorithm you're proposing here, but
>>> it sounds vaguely like what I'm writing now, so you might like the
>>> result.
>> Where/when can I find this code?
>
> You and others can now find it here: https://debbugs.gnu.org/41572#186

That thread has a latest patch that allows marker files to designate
top-level projects: I'm not interested in using marker files
specifically, so I don't see the relevance.  I can't deduce from the
patch what you're trying to solve or if it solves my use case.

Again, I'm interested in being able to designate projects that may only
exist inside other projects, and having a the C-x p family of commands
to allow a choice of inner project or any of the associated
super-projects.

There are maybe multiple ways to go about this: I've suggested what I
think is a promising one.

João

Re: Subprojects in project.el

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 26/11/22 02:00, João Távora wrote:
>>> I don't entirely understand the algorithm you're proposing here, but
>>> it sounds vaguely like what I'm writing now, so you might like the
>>> result.
>> Where/when can I find this code?
>
> In the bug report I've linked to a few times here. I've Cc'd you.

I've seen your patch, and I couldn't understand how it solves what I'm
looking for.  Replied in other threads.

Re: Subprojects in project.el

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 26/11/22 02:37, João Távora wrote:
>> Dmitry Gutov <dgutov@yandex.ru> writes:
>> 
>>>>>> I can't understand what is discourteous about this.
>>>>> That would be not following the procedure the maintainer has asked you
>>>>> to follow.
>>>> If that means silencing me on emacs-devel, then you're out of luck.
>>>
>>> Is that what you do when you ask somebody to use the bug tracker?
>> I'll use the bug tracker when I think it's appropriate.  Let's not
>> insinuate I'm some kind of inconsiderate delinquent for not moving the
>> discussion there as you would want.  I'm not reporting a bug and I've
>> politely declined your suggestion, so stop beating this horse.
>
> Must be nice to be the person who gets to decide what is appropriate
> in any situation.

As long as this list's maintainer doesn't object, I get to decide where
_I_ post to, thank you very much.

> By modifying each and every command. I don't think it would be
> appropriate for 'project-current' itself to react to the value of
> current-prefix-arg.

It's very unfortunate to modify "each and every command" unless of
course you mean doing it via a uniform interface.

I don't understand why these "project" commands that operate on a
project don't befittingly take a PROJECT argument.  That argument's
value could be interactively calculated from a
project-read-project-maybe command that decides if and how to prompt.

>>> What's missing in the infrastructure?
>> Not much, I would say.  But I think at least:
>> * A way that I can add an element to project-find-functions that
>>    understands that a super-project has been detected already in the
>>    current search and proceeds to find sub-projects inside it.  This is
>>    what I posted code for.
>> * A way for M-- (or similar) to consistently affect all (or most) of
>> the
>>    operations in the C-x p keymap so that we can choose if the operation
>>    operates on the super-project, if it exists.  Unfortunately some of
>>    these commands (like project-find-files) already take a prefix
>>    argument to mean different things.  But it's not too bad.
>> * A new project type, similar to the '(transient . "dir") project
>> (and
>>    inheriting most of its operations) that also keeps a record of the
>>    super-project found.  This might not be strictly necessary, but could
>>    come in handy later for efficiency reasons.
>
> Sounds pretty complicated. See if the latest patch solves your
> immediate problems just as well.

In your patch to be cramming it all into the VC type (which the comment
itself admits becomes "VC and etc.") and trying very hard not to create
a new subproject type.  But if you did that you could easily e.g. reuse
the super-project's ignore rules etc in the sub-project.

João

Re: Subprojects in project.el

[Dmitry Gutov]

On 26/11/22 04:59, Stefan Monnier wrote:
> I suggest we choose two different names for those two different cases.

Suggestions welcome.

Subprojects vs inner projects?

Re: Subprojects in project.el

[Dmitry Gutov]

On 26/11/22 11:42, João Távora wrote:
> Dmitry Gutov <dgutov@yandex.ru> writes:
> 
>> On 26/11/22 02:37, João Távora wrote:
>>> Dmitry Gutov <dgutov@yandex.ru> writes:
>>>
>>>>>>> I can't understand what is discourteous about this.
>>>>>> That would be not following the procedure the maintainer has asked you
>>>>>> to follow.
>>>>> If that means silencing me on emacs-devel, then you're out of luck.
>>>>
>>>> Is that what you do when you ask somebody to use the bug tracker?
>>> I'll use the bug tracker when I think it's appropriate.  Let's not
>>> insinuate I'm some kind of inconsiderate delinquent for not moving the
>>> discussion there as you would want.  I'm not reporting a bug and I've
>>> politely declined your suggestion, so stop beating this horse.
>>
>> Must be nice to be the person who gets to decide what is appropriate
>> in any situation.
> 
> As long as this list's maintainer doesn't object, I get to decide where
> _I_ post to, thank you very much.

I'll keep that in mind.

>> By modifying each and every command. I don't think it would be
>> appropriate for 'project-current' itself to react to the value of
>> current-prefix-arg.
> 
> It's very unfortunate to modify "each and every command" unless of
> course you mean doing it via a uniform interface.
> 
> I don't understand why these "project" commands that operate on a
> project don't befittingly take a PROJECT argument.  That argument's
> value could be interactively calculated from a
> project-read-project-maybe command that decides if and how to prompt.

Hysterical raisins.

But 'C-x p f' and 'C-x p g' call 'project-current' directly. What's that 
about some new command?

>>>> What's missing in the infrastructure?
>>> Not much, I would say.  But I think at least:
>>> * A way that I can add an element to project-find-functions that
>>>     understands that a super-project has been detected already in the
>>>     current search and proceeds to find sub-projects inside it.  This is
>>>     what I posted code for.
>>> * A way for M-- (or similar) to consistently affect all (or most) of
>>> the
>>>     operations in the C-x p keymap so that we can choose if the operation
>>>     operates on the super-project, if it exists.  Unfortunately some of
>>>     these commands (like project-find-files) already take a prefix
>>>     argument to mean different things.  But it's not too bad.
>>> * A new project type, similar to the '(transient . "dir") project
>>> (and
>>>     inheriting most of its operations) that also keeps a record of the
>>>     super-project found.  This might not be strictly necessary, but could
>>>     come in handy later for efficiency reasons.
>>
>> Sounds pretty complicated. See if the latest patch solves your
>> immediate problems just as well.
> 
> In your patch to be cramming it all into the VC type (which the comment
> itself admits becomes "VC and etc.") and trying very hard not to create
> a new subproject type.  But if you did that you could easily e.g. reuse
> the super-project's ignore rules etc in the sub-project.

How does one "easily reuse the super-project's ignore rules"? Would the 
sub-project type be called "sub-project"?

What if one VC repository is nested inside another? What if one "plain" 
project is nested inside another?

Re: Subprojects in project.el

[Dmitry Gutov]

On 26/11/22 11:23, João Távora wrote:
> Dmitry Gutov <dgutov@yandex.ru> writes:
> 
>> On 26/11/22 02:00, João Távora wrote:
> 
>> We aren't even close to top efficiency WRT file operations now. And
>> your idea is also based on hand-crafting project-find-functions to
>> only contain the necessary items.
> 
> project-find-functions is part of project.el's API.  So it makes a lot
> of sense to use, otherwise why would it be there?  We can provide some
> pre-baked functions to put there, or already put there for users.  The
> key point is that functions bail out early if they don't the adequate
> context.  One of them is the one I proposed earlier, which uses a local
> project-subproject-prefixes variable.  Another would search for marker
> files and consult a tramp-specific user variable for opt-out.

project-find-functions up until now has worked like many other hooks we 
have (e.g. completion-at-point-functions): the first element that 
returns non-nil is used, the others are not paid attention.

That allows these hooks to be used by major and minor modes, rather than 
having the user curate the full list. As long as priorities are set 
correctly, the system works. E.g. projectile-mode can just add its own 
element at the beginning, and not worry about stuff that already exists 
there.

>> As opposed to real-life scenario of Projectile just adding its
>> function at the front.
>>
>>>> I don't entirely understand the algorithm you're proposing here, but
>>>> it sounds vaguely like what I'm writing now, so you might like the
>>>> result.
>>> Where/when can I find this code?
>>
>> You and others can now find it here: https://debbugs.gnu.org/41572#186
> 
> That thread has a latest patch that allows marker files to designate
> top-level projects: I'm not interested in using marker files
> specifically, so I don't see the relevance.  I can't deduce from the
> patch what you're trying to solve or if it solves my use case.

Previously you said:

 > Fwiw, Danny's tip is what I used at a recent day job gig for defining 
subprojects in a 400k files repo.

where Danny's snippet looked like:

   (defun project-find-project.el (dir)
     "Returns a `manual-project' instance if the project of the current
   DIR has a .project.el file in its root directory."
     (let ((root (locate-dominating-file dir ".project.el")))
       (when root
         (cons 'transient root))))

   (add-hook 'project-find-functions #'project-find-project.el)

...but you didn't use marker files? Or you don't want to use them 
anymore? I'm really confused here.

Re: Subprojects in project.el

[Stefan Monnier]

Dmitry Gutov [2022-11-26 14:30:50] wrote:
> On 26/11/22 04:59, Stefan Monnier wrote:
>> I suggest we choose two different names for those two different cases.
> Suggestions welcome.

I couldn't think of any when I wrote the above :-(

> Subprojects vs inner projects?

Hmm... "inner project" doesn't give me much intuition for what it is.
Maybe that's a good thing (forces me to learn what it means rather than
assume I know what it's about).

"nested project" is another option that comes to mind in response to
the above.  Not sure if it clearly enough conveys the distinction
between the two cases, tho.


        Stefan

Re: Subprojects in project.el

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1206 bytes --]

On Sat, Nov 26, 2022, 13:34 Dmitry Gutov <dgutov@yandex.ru> wrote:

> On 26/11/22 11:23, João Távora wrote:
> > Dmitry Gutov <dgutov@yandex.ru> writes:
> >
> .
>
> Previously you said:
>
>  > Fwiw, Danny's tip is what I used at a recent day job gig for defining
> subprojects in a 400k files repo.
>
> where Danny's snippet looked like:
>
>    (defun project-find-project.el (dir)
>      "Returns a `manual-project' instance if the project of the current
>    DIR has a .project.el file in its root directory."
>      (let ((root (locate-dominating-file dir ".project.el")))
>        (when root
>          (cons 'transient root))))
>
>    (add-hook 'project-find-functions #'project-find-project.el)
>
> ...but you didn't use marker files? Or you don't want to use them
> anymore? I'm really confused here.
>

Don't be confused, this is not hard.  I used code similar to Danny's,
meaning i used a similar function that uses eglot-lsp-context (which you
omitted) but it uses simply path names. I can't use marker files in that
large project nor do I want or need to.  I don't have a way to choose the
subject of project operations, hence all this discussion.

João

>

[-- Attachment #2: Type: text/html, Size: 1996 bytes --]

Re: Subprojects in project.el

[Dmitry Gutov]

On 26/11/22 19:32, Stefan Monnier wrote:
> "nested project" is another option that comes to mind in response to
> the above.  Not sure if it clearly enough conveys the distinction
> between the two cases, tho.

I like "nested projects", thanks. That's probably as distinctive as 
we're ever going to get.

Re: Subprojects in project.el

[Juri Linkov]

> Anyway, indeally I want these three main operations (find-file, grep,
> compile) to run in the inner sub-project by default.  By typing
> something more, like, say, a negative prefix argument, I want to be able
> to be given the possibility to operate on the super-project instead.

Or generally a numeric prefix argument could define the depth of the
nested project to use.  Then every level could set own root, e.g.:

  (dir-locals-set-class-variables 'project-root ((nil . (project-root t))))
  (dir-locals-set-directory-class "~/Source/very-big-project" 'project-root)
  (dir-locals-set-directory-class "~/Source/very-big-project/foo" 'project-root)
  (dir-locals-set-directory-class "~/Source/very-big-project/foo/bar" 'project-root)

While using e.g. 'C-x p g' in "~/Source/very-big-project/foo/bar"
by default will use the closest root, i.e. the same directory,
but 'M-1 C-x p g' will use the root "~/Source/very-big-project/foo",
and 'M-2 C-x p g' will use the root "~/Source/very-big-project/".

Re: Subprojects in project.el

[Dmitry Gutov]

On 27/11/22 21:25, Juri Linkov wrote:
>> Anyway, indeally I want these three main operations (find-file, grep,
>> compile) to run in the inner sub-project by default.  By typing
>> something more, like, say, a negative prefix argument, I want to be able
>> to be given the possibility to operate on the super-project instead.
> Or generally a numeric prefix argument could define the depth of the
> nested project to use.  Then every level could set own root, e.g.:
> 
>    (dir-locals-set-class-variables 'project-root ((nil . (project-root t))))
>    (dir-locals-set-directory-class "~/Source/very-big-project" 'project-root)
>    (dir-locals-set-directory-class "~/Source/very-big-project/foo" 'project-root)
>    (dir-locals-set-directory-class "~/Source/very-big-project/foo/bar" 'project-root)
> 
> While using e.g. 'C-x p g' in "~/Source/very-big-project/foo/bar"
> by default will use the closest root, i.e. the same directory,
> but 'M-1 C-x p g' will use the root "~/Source/very-big-project/foo",
> and 'M-2 C-x p g' will use the root "~/Source/very-big-project/".

If you like, sure.

Re: Subprojects in project.el

[Dmitry Gutov]

On 26/11/22 01:55, João Távora wrote:

>> In most cases we will say it's not a good idea, but when a practical
>> goal is described we will decide to either say "go ahead, it's okay in
>> this case", or, hopefully, suggest a different way to reach that goal
>> (just like I did with the 'project-parent' definition example). Or
>> rethink and throw away the whole design (hopefully not).
> 
> I don't think it has to be so extreme.  I don't understand why there
> isn't a user-callable construtor for a type of project that is currently
> represented by the '(transient . "<dir>") implementation detail.
> Demanding that the users of looking to add to project-find-functions
> additionally define a whole new type and all the operations to go with
> it is unreasonable, IMO.  I don't think it's "throwing away the whole
> design" to provide one such constructor or a means to simplify this.

Do you actually want to use 'transient' in more places?

I get why it's okay for Eglot: it only needs the root and the primitive 
buffer listing logic, and not anything else.

But instantiating it in a function that returns a value to be used for 
all (project.el related) purposes would not be wise.

> Even better, provide a CLOS class, so that users may subclass it and use
> inheritance in the CLOS generics.

CLOS classes (or cl-struct) codify the structure, though, forcing us to 
make it stable.

As long as we're talking about the 'transient' type, it will likely 
remain as-is for all the years to come, because there's little to be 
gathered from one user prompt aside from the directory (and that's the 
main place where the 'transient' instances come from).

So I'm not sure why you'd want a CLOS hierarchy start from it: there is 
nothing to inherit. Just create your classes. The very top one will have 
to have a definition (an obvious one, 2 line long) for 'project-root', 
and the result will have all the behaviors of 'transient' already.

Re: Subprojects in project.el (Was: Eglot, project.el, and python virtual environments)

[Tim Cross]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 25/11/22 00:46, Tim Cross wrote:
>> João Távora <joaotavora@gmail.com> writes:
>> 
>>> On Thu, Nov 24, 2022 at 3:01 AM Dmitry Gutov <dgutov@yandex.ru> wrote:
>>>
>>>   
>>>>   I'm imagining that traversing a directory tree with an arbitrary
>>>>   predicate is going to be slow. If the predicate is limited somehow (e.g.
>>>>   to a list of "markers" as base file name, or at least wildcards), 'git
>>>>   ls-files' can probably handle this, with certain but bounded cost.
>> I've seen references to superior performance benefits of git ls-file a
>> couple of times in this thread, which has me a little confused.
>> There has been lots in other threads regarding the importance of not
>> relying on and not basing development on an underlying assumption
>> regarding the VCS being used. For example, I would expect project.el to
>> be completely neutral with respect to the VCS used in a project.
>
> That's the situation where we can optimize this case: when a project is Git/Hg.
>
>> So how is git ls-file at all relevant when discussing performance
>> characteristics when identifying files in a project?
>
> Not files, though. Subprojects. Meaning, listing all (direct and indirect) subdirectories
> which satisfy a particular predicate. If the predicate is simple (has a particular project
> marker: file name or wildcard), it can be fetched in one shell command, like:
>
> git ls-files -co -- "Makefile" "package.json"
>
> (which will traverse the directory tree for you, but will also use Git's cache).
>
> If the predicate is arbitrary (i.e. implemented in Lisp), the story would become harder.
>
>> I also wonder if some of the performance concerns may be premature. I've
>> seen references to poor performance in projects with 400k or even 100k
>> files. What is the expected/acceptable performance for projects of that
>> size? How common are projects of that size? When considering
>> performance, are we not better off focusing on the common case rather
>> than extreme cases, leaving the extremes for once we have a known
>> problem we can then focus in on?
>
> OT1H, large projects are relatively rare. OT2H, having a need for subprojects seems to be
> correlated with working on large projects.
>
> What is the common case, in your experience, and how is it better solved? Globally
> customizing a list of "markers", or customizing a list of subprojects for every "parent"
> project?

In my personal experience, sub-projects have been more about project
structure and not size. I would agree they are more prevalent in large
projects, but can exist in medium and even smaller projects.

I don't think I have a preference for customizing a list of markers or a
list of sub project definitions per project. I suspect different
approaches will work better in different scenarios and neither is a
clear 'winner'. However, as pointed out by Stephan, terminology
confusion/meaning may well be contributing to the confusion here. Not
only am I unsure everyone is thinking the same thing when talking about
sub-projects, I'm not sure everyone is even talking about the same thing
when referencing 'project'.

I wrote a lot about how I use projects and sub-projects in my work flow
and then realised it probably isn't helping that much. It struck me that
perhaps the issue is that the notion of sub-projects isn't really that
useful in itself and may actually be more detrimental than useful.

When you think about it, a sub-project is really just a more narrow
project focus. A project is really just a collection of files and
environment settings which can be considered, for some purpose, as a
'unit' in itself. It might define the set of files used when considering
find and replace for a symbol, when looking for symbol completion
candidates, or file/buffer switching, opening, linting, cross
referencing etc. It may correspond to a VCS repository, but it may
not. It could cut across repositories, or it could be made up of
multiple repositories or it could simply be some bespoke virtual project
concept specific to a particular use case.

I guess what I want is the ability to define arbitrary collections of
files and environment settings as a project, have a way to select/target
a project and an API which various tools can use to get the files or
environment settings to then operate on. Whether one project can be
considered a sub-project of another project is less relevant compared to
the ability to select/identify the target project. Automatic definition
of projects based on VCS repositories is great and a real time saver,
but the ability to define what makes up a project manually is also
important. The ability of the system to automatically determine which
project is 'active' (for example, based on the location of the file
being opened) is good and having the system prompt you when it isn't
clear or when there are multiple options is useful, but just being able
to run a command to set the current project would also be
sufficient. However, how one project relates to another project i.e. sub
project, main project, etc, seem of limited use compared to just having
the ability to select a sub-set of the files and environment settings of
a project, whether we call these sub project or nested projects or
whatever, seems of limited benefit.

Re: Subprojects in project.el (Was: Eglot, project.el, and python virtual environments)

[Dmitry Gutov]

On 28/11/2022 06:10, Tim Cross wrote:

>> OT1H, large projects are relatively rare. OT2H, having a need for subprojects seems to be
>> correlated with working on large projects.
>>
>> What is the common case, in your experience, and how is it better solved? Globally
>> customizing a list of "markers", or customizing a list of subprojects for every "parent"
>> project?
> 
> In my personal experience, sub-projects have been more about project
> structure and not size. I would agree they are more prevalent in large
> projects, but can exist in medium and even smaller projects.

Sure.

> I don't think I have a preference for customizing a list of markers or a
> list of sub project definitions per project. I suspect different
> approaches will work better in different scenarios and neither is a
> clear 'winner'. However, as pointed out by Stephan, terminology
> confusion/meaning may well be contributing to the confusion here. Not
> only am I unsure everyone is thinking the same thing when talking about
> sub-projects, I'm not sure everyone is even talking about the same thing
> when referencing 'project'.

I suppose another way to look at it is, subprojects could be defined by 
technical boundaries (e.g. this dir is a frontend SPA, and that dir is 
our backend), or by social (we have a monorepo, but this dir belongs to 
our team).

The former approach should be easier to support reliably using markers, 
and the latter -- not necessarily so.

But I'd be happy to find out that 98% of our users' cases can be handled 
with markers.

> I wrote a lot about how I use projects and sub-projects in my work flow
> and then realised it probably isn't helping that much. It struck me that
> perhaps the issue is that the notion of sub-projects isn't really that
> useful in itself and may actually be more detrimental than useful.

It all depends on the individual workflows, for sure.

> When you think about it, a sub-project is really just a more narrow
> project focus. A project is really just a collection of files and
> environment settings which can be considered, for some purpose, as a
> 'unit' in itself. It might define the set of files used when considering
> find and replace for a symbol, when looking for symbol completion
> candidates, or file/buffer switching, opening, linting, cross
> referencing etc. It may correspond to a VCS repository, but it may
> not. It could cut across repositories, or it could be made up of
> multiple repositories or it could simply be some bespoke virtual project
> concept specific to a particular use case.

Enviroment settings -- we do not support (yet?). But depending on the 
person and the goal, dir-locals.el can help quite well.

Regarding cutting across repositories, etc, there is a line for cases 
wre can easily (or with minimal customization) support ooutb, with 
auto-detection that a lot of the users prefer. Versus very custom shapes 
which require one to write a custom backend.

But if someone has a particularly handy way to define an 
arbitrarily-shaped project, they can submit that backend for inclusion 
as well. We could call it "free-form", or something.

> I guess what I want is the ability to define arbitrary collections of
> files and environment settings as a project, have a way to select/target
> a project and an API which various tools can use to get the files or
> environment settings to then operate on. Whether one project can be
> considered a sub-project of another project is less relevant compared to
> the ability to select/identify the target project. Automatic definition
> of projects based on VCS repositories is great and a real time saver,
> but the ability to define what makes up a project manually is also
> important. The ability of the system to automatically determine which
> project is 'active' (for example, based on the location of the file
> being opened) is good and having the system prompt you when it isn't
> clear or when there are multiple options is useful, but just being able
> to run a command to set the current project would also be
> sufficient. However, how one project relates to another project i.e. sub
> project, main project, etc, seem of limited use compared to just having
> the ability to select a sub-set of the files and environment settings of
> a project, whether we call these sub project or nested projects or
> whatever, seems of limited benefit.

Regarding the latter, there are a couple of requests now to be able to 
run a particular action (project-find-file or project-find-regexp) 
against either the current project or the "parent" project (determined 
by the locations of the root dirs), based on the prefix argument. That 
seems reasonable enough.

Re: Subprojects in project.el

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

> But I'd be happy to find out that 98% of our users' cases can be
> handled with markers.

I think you'll begin to lose that bet right here in the Emacs repo.
Take the doc/ directory where Emacs's manuals live.  It's pretty
reasonable to consider a non-programmer (or non-Elisp) contributor
working on that directory almost exclusively, perhaps as a proof-reader.

That person would like C-x p f, C-x p g, etc to operate on the /doc
hierarchy almost always.  For example to escape the enormous amount of
grep matches in the Changelog files.  Occasionally, that person might
want to grep the super-project (Emacs itself) or run a compilation
command from there, but that would be infrequent.

Now, using marker files to mark doc/ as a subproject is not only
impossible (there are none) and also useless.

Marker files _are_ useful in e.g. large Javascript monorepos where a lot
of NPM packages live.  These are frequently added, removed, reshuffled.
There, looking for `package.json` to designate subprojects makes a lot
of sense, this is beyond question IMO.  But that doesn't make the case I
describe above an exotic exception in any way.

João

Re: Subprojects in project.el

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 26/11/22 11:42, João Távora wrote:
>> Dmitry Gutov <dgutov@yandex.ru> writes:
>> 
>>> On 26/11/22 02:37, João Távora wrote:
>>>> Dmitry Gutov <dgutov@yandex.ru> writes:
>>>>
>>>>>>>> I can't understand what is discourteous about this.
>>>>>>> That would be not following the procedure the maintainer has asked you
>>>>>>> to follow.
>>>>>> If that means silencing me on emacs-devel, then you're out of luck.
>>>>>
>>>>> Is that what you do when you ask somebody to use the bug tracker?
>>>> I'll use the bug tracker when I think it's appropriate.  Let's not
>>>> insinuate I'm some kind of inconsiderate delinquent for not moving the
>>>> discussion there as you would want.  I'm not reporting a bug and I've
>>>> politely declined your suggestion, so stop beating this horse.
>>>
>>> Must be nice to be the person who gets to decide what is appropriate
>>> in any situation.
>> As long as this list's maintainer doesn't object, I get to decide
>> where
>> _I_ post to, thank you very much.
>
> I'll keep that in mind.
>
>>> By modifying each and every command. I don't think it would be
>>> appropriate for 'project-current' itself to react to the value of
>>> current-prefix-arg.
>> It's very unfortunate to modify "each and every command" unless of
>> course you mean doing it via a uniform interface.
>> I don't understand why these "project" commands that operate on a
>> project don't befittingly take a PROJECT argument.  That argument's
>> value could be interactively calculated from a
>> project-read-project-maybe command that decides if and how to prompt.
>
> Hysterical raisins.
>
> But 'C-x p f' and 'C-x p g' call 'project-current' directly. What's
> that about some new command?

Don't know what you are talking aboutIt seems very reasonable to do
this instead:

diff --git a/lisp/progmodes/project.el b/lisp/progmodes/project.el
index c7b2c386ccd..02f5d25e054 100644
--- a/lisp/progmodes/project.el
+++ b/lisp/progmodes/project.el
@@ -1151,11 +1151,11 @@ project-compilation-buffer-name-function
                  (function :tag "Custom function")))
 
 ;;;###autoload
-(defun project-compile ()
+(defun project-compile (project)
   "Run `compile' in the project root."
   (declare (interactive-only compile))
-  (interactive)
-  (let ((default-directory (project-root (project-current t)))
+  (interactive (project-current t))
+  (let ((default-directory (project-root project))
         (compilation-buffer-name-function
          (or project-compilation-buffer-name-function
              compilation-buffer-name-function)))

Re: Subprojects in project.el

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

>> Hysterical raisins.
>>
>> But 'C-x p f' and 'C-x p g' call 'project-current' directly. What's
>> that about some new command?

> Don't know what you are talking aboutIt seems very reasonable to do
> this instead:

>diff --git a/lisp/progmodes/project.el b/lisp/progmodes/project.el
>index c7b2c386ccd..02f5d25e054 100644
>--- a/lisp/progmodes/project.el
>+++ b/lisp/progmodes/project.el
>@@ -1151,11 +1151,11 @@ project-compilation-buffer-name-function
>                  (function :tag "Custom function")))
> 
> ;;;###autoload
>-(defun project-compile ()
>+(defun project-compile (project)
>   "Run `compile' in the project root."
>   (declare (interactive-only compile))
>-  (interactive)
>-  (let ((default-directory (project-root (project-current t)))
>+  (interactive (project-current t))

Sorry I hit "send" before I wanted to.  This line of course be

  (interactive (list (project-current t)))

And the docstring should be

  "Run `compile' in PROJECT's root."

'project-current', when called with MAYBE-PROMPT, is running in an
interactive setting.  It could then consult the value of the prefix
argument and decide to prompt the user about which project to use
(including super-projects).

You can use this technique to a add PROJECT argument to all the other
relevant project-* operations without breaking backward compatibility,
even for non-interactive Lisp uses.

In cases where a "P" prefix argument is already in use, like C-x p f,
we'd have to repurpose the "negative prefix" part to _not_ mean
INCLUDE-ALL.  You'd affect only the people who are currently using M--
C-x p f instead of C-u C-x p f.

João

Re: Subprojects in project.el

[Dmitry Gutov]

On 29/11/2022 11:56, João Távora wrote:
> Dmitry Gutov <dgutov@yandex.ru> writes:
> 
>> But I'd be happy to find out that 98% of our users' cases can be
>> handled with markers.
> 
> I think you'll begin to lose that bet right here in the Emacs repo.
> Take the doc/ directory where Emacs's manuals live.  It's pretty
> reasonable to consider a non-programmer (or non-Elisp) contributor
> working on that directory almost exclusively, perhaps as a proof-reader.

Setting aside the artificial nature of this example, they could add 
"lispintro" or "lispref" to project-vc-extra-root-markers and have that 
nested project recognized.

As long as the directory contains at least some unique (in the scope of 
the containing project) file or directory, the marker-based approach can 
work fine.

> That person would like C-x p f, C-x p g, etc to operate on the /doc
> hierarchy almost always.  For example to escape the enormous amount of
> grep matches in the Changelog files.  Occasionally, that person might
> want to grep the super-project (Emacs itself) or run a compilation
> command from there, but that would be infrequent.

Sounds fine to me.

> Now, using marker files to mark doc/ as a subproject is not only
> impossible (there are none) and also useless.
> 
> Marker files _are_ useful in e.g. large Javascript monorepos where a lot
> of NPM packages live.  These are frequently added, removed, reshuffled.
> There, looking for `package.json` to designate subprojects makes a lot
> of sense, this is beyond question IMO.  But that doesn't make the case I
> describe above an exotic exception in any way.

I don't know how many people actually intend to do what you described, 
though. But it seems workable just the same.

Re: Subprojects in project.el

[Dmitry Gutov]

On 29/11/2022 12:17, João Távora wrote:
> Sorry I hit "send" before I wanted to.  This line of course be
> 
>    (interactive (list (project-current t)))
> 
> And the docstring should be
> 
>    "Run `compile' in PROJECT's root."
> 
> 'project-current', when called with MAYBE-PROMPT, is running in an
> interactive setting.  It could then consult the value of the prefix
> argument and decide to prompt the user about which project to use
> (including super-projects).
> 
> You can use this technique to a add PROJECT argument to all the other
> relevant project-* operations without breaking backward compatibility,
> even for non-interactive Lisp uses.

What do you mean, without breaking compatibility? Adding a new required 
argument means it will break any Lisp code that calls this function.

At least that's how I understand our standard criteria for this thing.

And it's probably not that big a deal if we had no other choice, but we 
do. Update the commands to call a different new function. That will 
actually require *fewer* changes, as you might have noticed now.

> In cases where a "P" prefix argument is already in use, like C-x p f,
> we'd have to repurpose the "negative prefix" part to_not_  mean
> INCLUDE-ALL.  You'd affect only the people who are currently using M--
> C-x p f instead of C-u C-x p f.

This hints that the choice of whether to use the parent or not should be 
made inside the command implementation. Perhaps inside the 'interactive' 
form, but not inside 'project-current'.

The principle of separation of concerns suggests the same.

Something like:

diff --git a/lisp/progmodes/project.el b/lisp/progmodes/project.el
index c1245704bcc..9e5fcfe0b06 100644
--- a/lisp/progmodes/project.el
+++ b/lisp/progmodes/project.el
@@ -1139,6 +1139,19 @@ project-prefixed-buffer-name
            (downcase mode)
            "*"))

+(defun project-parent (project)
+  (project-current nil (file-name-as-directory
+                        (file-name-directory
+                         (directory-file-name (project-root project))))))
+
+(defun project-current-or-parent (lookup-parent)
+  (let ((project (project-current t)))
+    (when lookup-parent
+      (setq project (project-parent project)))
+    (unless project
+      (user-error "No parent project"))
+    project))
+
  (defcustom project-compilation-buffer-name-function nil
    "Function to compute the name of a project compilation buffer.
  If non-nil, it overrides `compilation-buffer-name-function' for
@@ -1155,7 +1168,8 @@ project-compile
    "Run `compile' in the project root."
    (declare (interactive-only compile))
    (interactive)
-  (let ((default-directory (project-root (project-current t)))
+  (let ((default-directory (project-root (project-current-or-parent
+                                          (eq current-prefix-arg '-))))
          (compilation-buffer-name-function
           (or project-compilation-buffer-name-function
               compilation-buffer-name-function)))

Re: Subprojects in project.el

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

> On 29/11/2022 11:56, João Távora wrote:
>> Dmitry Gutov <dgutov@yandex.ru> writes:
>> 
>>> But I'd be happy to find out that 98% of our users' cases can be
>>> handled with markers.
>> I think you'll begin to lose that bet right here in the Emacs repo.
>> Take the doc/ directory where Emacs's manuals live.  It's pretty
>> reasonable to consider a non-programmer (or non-Elisp) contributor
>> working on that directory almost exclusively, perhaps as a proof-reader.
>
> Setting aside the artificial nature of this example, they could add
> "lispintro" or "lispref" to project-vc-extra-root-markers and have
> that nested project recognized.

Talk about artificial.  I don't want to say that "lispintro" or
"lispref" anywhere in my filesystem marks the doc/ subproject.  I might
have identically named elsewhere.  And these are much more likely to
change than doc/ won't.  In other words, it's not correct to describe a
subproject in terms of its interior structure.

And there's also the obvious drawback, that you yourself raised, that
looking for marker files is needlessly taxing in terms of file system
operations.  Especially, as you highlighted, under TRAMP or a
slow-to-access file systems.

> As long as the directory contains at least some unique (in the scope
> of the containing project) file or directory, the marker-based
> approach can work fine.

That's not what marker files are good at.  Markers, like the .git
directory, are great for identifying many projects whose root location
is not easily predictable but which have a very stable file element at
their root at all times.

But they're _not_ great for designating a fixed and easily predictable
subset of hierarchies in a within a more complex, longstanding, legacy
hierarchy.

> I don't know how many people actually intend to do what you described,
> though. But it seems workable just the same.

Whoever wants to reap the benefits of subprojects for NPM packages in a
monorepo with marker files will probably come across situations where
they want to reap the same benefits for fixed directories but without
marker files.  And vice versa, of course.  The solution to be adpted
should not favour marker files over other approaches.  And it's so
simple not to incur in this design mistake that's it's a bit baffling
that you keep insisting to do it.

João

Re: Subprojects in project.el

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

> What do you mean, without breaking compatibility? Adding a new
> required argument means it will break any Lisp code that calls this
> function.

The example I gave you was for a function explicitly marked "interactive
only", so it's not a problem.  In fact all these functions should be
"interactive only".  But in other cases, make that argument optional.

> And it's probably not that big a deal if we had no other choice, but
> we do. Update the commands to call a different new function. 

As I explain below, you'll be littering the code with non-idiomatic
Elisp.  

> That will actually require *fewer* changes, as you might have noticed
> now.

Not sure about that.

> diff --git a/lisp/progmodes/project.el b/lisp/progmodes/project.el
> index c1245704bcc..9e5fcfe0b06 100644
> --- a/lisp/progmodes/project.el
> +++ b/lisp/progmodes/project.el
> @@ -1139,6 +1139,19 @@ project-prefixed-buffer-name
>            (downcase mode)
>            "*"))
>
> +(defun project-parent (project)
> +  (project-current nil (file-name-as-directory
> +                        (file-name-directory
> +                         (directory-file-name (project-root project))))))
> +
> +(defun project-current-or-parent (lookup-parent)
> +  (let ((project (project-current t)))
> +    (when lookup-parent
> +      (setq project (project-parent project)))
> +    (unless project
> +      (user-error "No parent project"))
> +    project))
> +
>  (defcustom project-compilation-buffer-name-function nil
>    "Function to compute the name of a project compilation buffer.
>  If non-nil, it overrides `compilation-buffer-name-function' for
> @@ -1155,7 +1168,8 @@ project-compile
>    "Run `compile' in the project root."
>    (declare (interactive-only compile))
>    (interactive)
> -  (let ((default-directory (project-root (project-current t)))
> +  (let ((default-directory (project-root (project-current-or-parent
> +                                          (eq current-prefix-arg '-))))
>          (compilation-buffer-name-function
>           (or project-compilation-buffer-name-function
>               compilation-buffer-name-function)))

Sounds more complicated that it needs to be, and no reason why
project-current-or-parent shouldn't be just project-current, with
another optional argument at most.

Anyway my idea to add PROJECT is nothing exotic: it's the standard
idiomatic Elisp solution used in such basic functionality as M-x
display-buffer, find-file, etc.  There's no reason to stray from that.

João

Re: Subprojects in project.el

[Dmitry Gutov]

On 30/11/2022 00:21, João Távora wrote:
> Dmitry Gutov <dgutov@yandex.ru> writes:
> 
>> On 29/11/2022 11:56, João Távora wrote:
>>> Dmitry Gutov <dgutov@yandex.ru> writes:
>>>
>>>> But I'd be happy to find out that 98% of our users' cases can be
>>>> handled with markers.
>>> I think you'll begin to lose that bet right here in the Emacs repo.
>>> Take the doc/ directory where Emacs's manuals live.  It's pretty
>>> reasonable to consider a non-programmer (or non-Elisp) contributor
>>> working on that directory almost exclusively, perhaps as a proof-reader.
>>
>> Setting aside the artificial nature of this example, they could add
>> "lispintro" or "lispref" to project-vc-extra-root-markers and have
>> that nested project recognized.
> 
> Talk about artificial.  I don't want to say that "lispintro" or
> "lispref" anywhere in my filesystem marks the doc/ subproject.  I might
> have identically named elsewhere.  And these are much more likely to
> change than doc/ won't.  In other words, it's not correct to describe a
> subproject in terms of its interior structure.

You don't want to. Okay.

But updating the config to use a different marker is also not a long 
task. It's not like someone's going to go on renaming "lispinto" to some 
other name and back over and over during the project lifetime.

> And there's also the obvious drawback, that you yourself raised, that
> looking for marker files is needlessly taxing in terms of file system
> operations.  Especially, as you highlighted, under TRAMP or a
> slow-to-access file systems.

Only when the goal is to exclude the nested projects' files from being 
listed in the output of 'project-files' of the encompassing project. I 
asked you about this, and you answered that this is a non-goal.

Otherwise, the VC Project backend uses file markers already anyway 
(that's what vc-responsible-backend is based on). The latest proposals 
to make the list of markers configurable in bug#41572 actually make this 
logic faster, not slower. Especially over Tramp.

>> I don't know how many people actually intend to do what you described,
>> though. But it seems workable just the same.
> 
> Whoever wants to reap the benefits of subprojects for NPM packages in a
> monorepo with marker files will probably come across situations where
> they want to reap the same benefits for fixed directories but without
> marker files.  And vice versa, of course.  The solution to be adpted
> should not favour marker files over other approaches.  And it's so
> simple not to incur in this design mistake that's it's a bit baffling
> that you keep insisting to do it.

What's baffling is how you insist on not reading or trying the patches 
that's been available in bug#41572 -- some for several days, some for 
over a year, which implement both this and other approaches.

Re: Subprojects in project.el

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

> But updating the config to use a different marker is also not a long
> task. It's not like someone's going to go on renaming "lispinto" to
> some other name and back over and over during the project lifetime.

I can't get through to you that designating a project that is rooted
stably in a subdirectory based on one of its interior components, which
by definition is what changes more often in a subproject is a bad idea.
Or that a subdirectory "lispinto" exist multiple times in the same
hierarchy, or in other hierarchies and have completely other meanings.
Too bad.

>> And there's also the obvious drawback, that you yourself raised, that
>> looking for marker files is needlessly taxing in terms of file system
>> operations.  Especially, as you highlighted, under TRAMP or a
>> slow-to-access file systems.
>
> Only when the goal is to exclude the nested projects' files from being
> listed in the output of 'project-files' of the encompassing project. I
> asked you about this, and you answered that this is a non-goal.

No.  Simply making more unnecessary file system accesses is a non-goal.

> What's baffling is how you insist on not reading or trying the patches
> that's been available in bug#41572 -- some for several days, some for
> over a year, which implement both this and other approaches.

If that patch requires me to use marker files, as you keep suggesting,
there's no point.

In fact, there's no point in continuing this discusison.  It's very
clear you are set on not taking any suggestions, and are convinced the
use case I've described is somehow from an esoteric 2%.  So do whatever
you want.

João

Re: Subprojects in project.el

[Dmitry Gutov]

On 30/11/2022 02:54, João Távora wrote:
>> What's baffling is how you insist on not reading or trying the patches
>> that's been available in bug#41572 -- some for several days, some for
>> over a year, which implement both this and other approaches.
> If that patch requires me to use marker files, as you keep suggesting,
> there's no point.

:rolleyes:

Just read the fucking patch already.

Re: Subprojects in project.el

[Dmitry Gutov]

On 30/11/2022 00:52, João Távora wrote:
> Dmitry Gutov <dgutov@yandex.ru> writes:
> 
>> What do you mean, without breaking compatibility? Adding a new
>> required argument means it will break any Lisp code that calls this
>> function.
> 
> The example I gave you was for a function explicitly marked "interactive
> only", so it's not a problem.  In fact all these functions should be
> "interactive only".  But in other cases, make that argument optional.

The commands that are marked 'interactive-only' are those that we have 
decided are useless in Lisp programs. Others aren't marked such.

>> +(defun project-parent (project)
>> +  (project-current nil (file-name-as-directory
>> +                        (file-name-directory
>> +                         (directory-file-name (project-root project))))))
>> +
>> +(defun project-current-or-parent (lookup-parent)
>> +  (let ((project (project-current t)))
>> +    (when lookup-parent
>> +      (setq project (project-parent project)))
>> +    (unless project
>> +      (user-error "No parent project"))
>> +    project))
>> +
>>   (defcustom project-compilation-buffer-name-function nil
>>     "Function to compute the name of a project compilation buffer.
>>   If non-nil, it overrides `compilation-buffer-name-function' for
>> @@ -1155,7 +1168,8 @@ project-compile
>>     "Run `compile' in the project root."
>>     (declare (interactive-only compile))
>>     (interactive)
>> -  (let ((default-directory (project-root (project-current t)))
>> +  (let ((default-directory (project-root (project-current-or-parent
>> +                                          (eq current-prefix-arg '-))))
>>           (compilation-buffer-name-function
>>            (or project-compilation-buffer-name-function
>>                compilation-buffer-name-function)))
> 
> Sounds more complicated that it needs to be, and no reason why
> project-current-or-parent shouldn't be just project-current, with
> another optional argument at most.

If it's a new optional argument vs new function, I can see the benefits 
of either. OT1H, project-current is shorter.

OT2H, some code (or even a person) might like to check (fboundp 
'project-current-or-parent).

Anyway, I'm not crazy about the name of the argument (lookup-parent), so 
suggestions are welcome on that.

Re: Subprojects in project.el

[João Távora]

Dmitry Gutov <dgutov@yandex.ru> writes:

> Just read the fucking patch already.

I think you need to chill, and then go do whatever you want.  I'm out of
this thread, goodbye.

Re: Subprojects in project.el (Was: Eglot, project.el, and python virtual environments)

[Dmitry Gutov]

Hi again,

It's probably time to continue here.

On 26/11/2022 09:26, Eli Zaretskii wrote:

>>> Again, you are talking about project kinds already supported by project.el
>>> as its built-ins.  I'm asking how to produce a project of a custom type.
>>
>> To produce a project of custom type, you decide the data structure for
>> that type, write implementations for all the required (and perhaps some
>> optional) generic methods, then create that structure.
> 
> It seems I need to spell out everything every time I'm writing something
> because otherwise you interpret it out of context.  So here: I'm asking how
> to produce a customized project whose type is 'transient', but which doesn't
> go by the rules of the built-in 'transient' project.

I'm trying to be precise both to avoid misunderstanding. Even here you 
were using the terminology that seems to conflict with established practice.

IIUC, you want to produce a project whose type is not 'transient', but 
which inherits from 'transient' some of its behaviors.

> As for "project of custom type", where "custom" means "not one of the known
> types", there's still a situation where my project is similar enough to one
> of the built-in types to make reimplementation of all the APIs overkill.
> One could want to use some of the APIs without change and customize others.
> So this, too, is something that project.el should IMO allow without too much
> fuss.

So, what you are asking for is perfectly reasonable in theory. Also the 
current theory of OOP suggests that type inheritance comes with its 
downsides as well. For instance, the straightforward way Joao was 
suggesting (create a defclass) will mean nailing down the internal 
structure of a type which would make it more difficult to create 
internal changes to it -- because somebody might have inherited from it, 
and might be using the exposed fields, etc.

Asking for a public constructor, however, can help in this regard as 
well, if you understand how to implement delegation using it. Meaning, 
you would still create your own type, but then to delegate the 
implementations of methods you would create an instance of the type to 
delegate to, and call said methods on it. This the type's evolution 
would only be limited by the backward compatibility of its constructor.

>> You asked, however, how to instantiate a project of a type belonging to
>> "someone else". But didn't explain why.
> 
> The "why" is "because one needs to".

The question also sounded odd because there is little point to "inherit" 
from 'transient'. Because that type's definition is only two lines:

   (cl-defmethod project-root ((project (head transient)))
     (cdr project))

There is literally nothing to reuse. The rest of that type's behaviors 
reside in the default definitions of other generic functions. So you can 
just create your own type, create its method for the above generic 
function, and then override the rest of the behaviors as needed.

If you wanted to inherit from the 'VC-aware' backend, however, that is a 
different discussion. I would ask which parts of it you need, and 
whether you intend to honor the project-vc-* user options, as its 
methods do. Or whether perhaps you just wanted to reuse the VC-based 
fast file listing implementation. Which would be a little easier to make 
public with an established signature.

>> The reasons not to rely on internal structure, as a reminder:
>>
>> - The internal structures are prone to change, and you don't want your
>> code to break when that happens.
>> - When implementing some feature that works with projects, you generally
>> want it to work with all kinds of projects (which was the whole point of
>> project.el -- to make this approach work). And that would mean talking
>> to them through common methods, rather that examine the internals of
>> this or that project type.
> 
> I'm fine with all that, but it sounds like the above makes it impossible to
> implement custom project objects, maybe because there's no make-project
> method that one could use and/or subclass.

I usually answer this way because often it's not a good idea, and there 
are better ways to reach the goals of the person who's asking.

Still, if you do want to inherit from 'VC-aware', we can make a public 
constructor for it. Probably after Emacs 29 is released, and we can see 
that the structure is settled (it's looking this way now).

As for 'transient', well, the constructor can be added as well. The 
structure is stable and least likely to change; unless we just make an 
executive decision to turn them all into structs or whatever. I just 
didn't want to encourage people using it -- even Joao's usage is odd 
because he not only calls the 'project-root' function, but also 
'project-files', and it's just luck that its behavior suits his current 
goals.

Re: Subprojects in project.el (Was: Eglot, project.el, and python virtual environments)

[Eli Zaretskii]

> Date: Fri, 2 Dec 2022 03:32:42 +0200
> Cc: monnier@iro.umontreal.ca, danny@dfreeman.email, eric@ericabrahamsen.net,
>  emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> Still, if you do want to inherit from 'VC-aware', we can make a public 
> constructor for it. Probably after Emacs 29 is released, and we can see 
> that the structure is settled (it's looking this way now).
> 
> As for 'transient', well, the constructor can be added as well. The 
> structure is stable and least likely to change; unless we just make an 
> executive decision to turn them all into structs or whatever. I just 
> didn't want to encourage people using it -- even Joao's usage is odd 
> because he not only calls the 'project-root' function, but also 
> 'project-files', and it's just luck that its behavior suits his current 
> goals.

I'm just surprised that a simple request to be able to create a project type
that is not one of the 2 built-in types is not answered by a simple "use
this and that APIs".  project.el strives very hard to be generic, but what
is the use in doing that if extending it by 3rd-party code is so
complicated, and on top of that is not already available?

So yes, I think we do have public constructors and whatever else could be
reasonably needed if one wants to subclass either of the two built-in
project types.  For this purpose, I don't think it matters how rich the
built-in type is -- that is something for the sub-classing application to
worry about.  We just need to give them enough rope, and leave the rest up
to them, IMO.

Re: Subprojects in project.el (Was: Eglot, project.el, and python virtual environments)

[Dmitry Gutov]

On 02/12/2022 16:16, Eli Zaretskii wrote:
>> Date: Fri, 2 Dec 2022 03:32:42 +0200
>> Cc: monnier@iro.umontreal.ca, danny@dfreeman.email, eric@ericabrahamsen.net,
>>   emacs-devel@gnu.org
>> From: Dmitry Gutov <dgutov@yandex.ru>
>>
>> Still, if you do want to inherit from 'VC-aware', we can make a public
>> constructor for it. Probably after Emacs 29 is released, and we can see
>> that the structure is settled (it's looking this way now).
>>
>> As for 'transient', well, the constructor can be added as well. The
>> structure is stable and least likely to change; unless we just make an
>> executive decision to turn them all into structs or whatever. I just
>> didn't want to encourage people using it -- even Joao's usage is odd
>> because he not only calls the 'project-root' function, but also
>> 'project-files', and it's just luck that its behavior suits his current
>> goals.
> 
> I'm just surprised that a simple request to be able to create a project type
> that is not one of the 2 built-in types is not answered by a simple "use
> this and that APIs".  project.el strives very hard to be generic, but what
> is the use in doing that if extending it by 3rd-party code is so
> complicated, and on top of that is not already available?

But that's what defines a project type: its implementations of the 
generic functions.

> So yes, I think we do have public constructors and whatever else could be
> reasonably needed if one wants to subclass either of the two built-in
> project types.  For this purpose, I don't think it matters how rich the
> built-in type is -- that is something for the sub-classing application to
> worry about.  We just need to give them enough rope, and leave the rest up
> to them, IMO.

FWIW, the built-in types's structures have been fairly stable for a 
while. So for most practical purposes people should be able to "extend" 
them already, definitely if it's for personal use. But for public use as 
well, if the package author is willing to provide prompt updates in the 
case of potential (rare) breakage.

I just rather hear about actual cases or intended scenarios, to decide 
how to support them better. E.g. providing constructors will "stabilize" 
some stuff, but won't make things much easier, coding-wise.

Re: Subprojects in project.el (Was: Eglot, project.el, and python virtual environments)

[Dmitry Gutov]

On 02/12/2022 17:08, Dmitry Gutov wrote:
> project.el strives very hard to be generic, but what
> is the use in doing that if extending it by 3rd-party code is so
> complicated, and on top of that is not already available?

Speaking of "extending is complicated", I have shown the two-line 
definition of 'transient', right? A custom type can look just as short 
(or not; depending on its needs).

So it's more of a matter of understanding how the pieces fit.

For instance, using the new type will require plugging it into 
project-find-functions. And for the VC-aware type, its element is one of 
the more complex pieces of its implementation. Which makes sense: that's 
where domain complexity lies.

So how one would be able to reuse both it and the existing logic? Here's 
an example of overriding the 'project-name' with a new type without 
requiring a public constructor for the VC-aware type:

(defun project-try-my-type (dir)
   (let ((vc-instance (project-try-vc dir))))
   (cons 'my-type vc-instance))

(cl-defmethod project-root ((project (head my-type)))
   (project-root (cdr project)))

(cl-defmethod project-external-roots ((project (head my-type)))
   (project-external-roots (cdr project)))

(cl-defmethod project-files ((project (head my-type)) &optional dirs)
   (project-files (cdr project) dirs))

(cl-defmethod project-ignores ((project (head my-type)) dir)
   (project-ignores (cdr project) dir))

(cl-defmethod project-buffers ((project (head my-type)))
   (project-buffers (cdr project)))

(cl-defmethod project-name ((project (head my-type)))
   ;; Do something different here
   )

I suppose it does look a little verbose... A defclass or defstruct would 
allow to declare automatic inheritance between types. I wonder if there 
are any other options that don't imply structural inheritance, or if we 
can simply say "inheriting is fine, but please don't rely on the fields 
staying the same".

Re: Subprojects in project.el

[Stefan Monnier]

>> I'm just surprised that a simple request to be able to create a project
>> type
>> that is not one of the 2 built-in types is not answered by a simple "use
>> this and that APIs".  project.el strives very hard to be generic, but what
>> is the use in doing that if extending it by 3rd-party code is so
>> complicated, and on top of that is not already available?
>
> But that's what defines a project type: its implementations of the
>  generic functions.

Indeed.  AFAIK the general approach for APIs that rely on CLOS-style OO
programming is that all the classes are kept "abstract" and what you see
is only generic functions.  IOW the API is exclusively defined by the
generic functions.

So you create your own type any which way you want and then provide
implementations of the API's methods.


        Stefan

Re: Subprojects in project.el

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > :rolleyes:

  > Just read the fucking patch already.

Whatever the specific disagreement is, I'm sure it is possible to
discuss it in amicable terms.  Please try to apply the Kind
Communcation Guidelines to the way you post.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Subprojects in project.el

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1071 bytes --]

On Fri, Dec 2, 2022 at 3:44 PM Stefan Monnier <monnier@iro.umontreal.ca>
wrote:

> Indeed.  AFAIK the general approach for APIs that rely on CLOS-style OO
> programming is that all the classes are kept "abstract" and what you see
> is only generic functions.  IOW the API is exclusively defined by the
> generic functions.
>

Yes, and such an API is called a "protocol" in CLOS parlance.  But the
exclusivity you suggest isn't true: classes and inheritance are a big
part of CLOS and CLOS-style OO.  IME,  CLOS's power is only fully
realized when you use both.

In Emacs Lisp's flavour of CLOS,  generic functions have somewhat
esoteric  dispatching mechanisms that I've never seen or needed in a
CLOS system when using classes, inheritance, and method combinations.

So +1 for using classes in more Elisp packages.  See for example how
eglot.el subclasses jsonrpc.el's classes. It's not as easy as in Common
Lisp since we don't have things like the beautiful inspector of SLY/SLIME.
But classes still beat those funky cons by a mile IMHO.

João

[-- Attachment #2: Type: text/html, Size: 1829 bytes --]

Re: Subprojects in project.el

[Dmitry Gutov]

On 03/12/2022 00:47, Richard Stallman wrote:
> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
> 
>    > :rolleyes:
> 
>    > Just read the fucking patch already.
> 
> Whatever the specific disagreement is, I'm sure it is possible to
> discuss it in amicable terms.  Please try to apply the Kind
> Communcation Guidelines to the way you post.

You can read the whole exasperating discussion and see that I really tried.

Re: Subprojects in project.el

[Eli Zaretskii]

> Date: Sat, 3 Dec 2022 01:46:19 +0200
> Cc: joaotavora@gmail.com, theophilusx@gmail.com, monnier@iro.umontreal.ca,
>  danny@dfreeman.email, eric@ericabrahamsen.net, emacs-devel@gnu.org
> From: Dmitry Gutov <dgutov@yandex.ru>
> 
> > Whatever the specific disagreement is, I'm sure it is possible to
> > discuss it in amicable terms.  Please try to apply the Kind
> > Communcation Guidelines to the way you post.
> 
> You can read the whole exasperating discussion and see that I really tried.

When you get to the point where the only response is like what you posted,
my personal recommendation is not to respond at all.

Re: Subprojects in project.el (Was: Eglot, project.el, and python virtual environments)

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 3427 bytes --]

On Fri, Dec 2, 2022 at 1:33 AM Dmitry Gutov <dgutov@yandex.ru> wrote:

>
> So, what you are asking for is perfectly reasonable in theory. Also the
> current theory of OOP suggests that type inheritance comes with its
> downsides as well. For instance, the straightforward way Joao was
> suggesting (create a defclass) will mean nailing down the internal
> structure of a type which would make it more difficult to create
> internal changes to it -- because somebody might have inherited from it,
> and might be using the exposed fields, etc.
>

I'm sorry, but it is imperative that this misconception be dispelled.

Just by doing:

  (defclass project-foo-project ()
    ((impl-detail :initform 42))

One is in no way exposing that implementation detail to the
subclassing user (if one is concerned the user uses slot-value
to access impl-detail's value, one may name it with the '--'
double-dash convention or even make it a keyword, but I would say
that's completely unnecessary here)

If one creates a slot with an reader:

  (defclass project-foo()
    ((some-implementation-detail-slot :initform 42)
     (root :initarg :root :reader project-root)))

then one _does_ expose an interface, in that one is adding a
method to the generic function project-root, which is exactly
what is desired.  Likewise for :writer and :accessor.

So, long after someone has inherited from project-foo one may
as well erase 'root' and represent the root of a foo project in
Morse code or anything like that just by reimplementing the
project-root method for project-foo.

Thus, it's false that using defclass means "nailing down the internal
structure of a type".  It's just not true for CLOS (or any other OO
system I know, for that matter).

As Stefan highlighted earlier, the generic functions are the
protocol.  Offering classes and inheritance doesn't change that:
it only makes them more powerful and simpler to instantiate.

And if one wants to add some pre-, post- or around- processing
to the `project-root` method created by defclass, one need
only use :before, :after or :around specifiers.

In fact, even if one wanted to prevent instancing of a specific
class (I don't think we should here, but let's say project-foo is
abstract or a singleton) defclass is still the way to go.  One can use a
specialization of make-instance or a add a method to initialize-instance
for that.

In fact, the "nailing down" problem is precisely what the current
implementation promotes.  The strange technique (transient . "some/path")
used here doesn't actively prevent any user from instantiating it
because 'cons' and 'transient' are both perfectly accessible symbols.
Users
will likely use them to work around the missing constructor.  And once
that happens, _that's_ when `project.el` is stuck with that internal
implementation.  But if one uses defclass instead, one can retain this
freedom and even control or monitor the creation of objects with
make-instance and associated CLOS protocols  (initialize-instance,
reinitialize-instance, etc).

Have a look at how eglot-lsp-server class hierarchy to see
some of these concepts in action.  For more examples, I
recommend any good CLOS book.

Finally, I know eieio.el isnt CLOS: it doesn't perfectly replicate
CLOS unfortunately, but it's decently close for these basic use
cases in my personal experience.

João

[-- Attachment #2: Type: text/html, Size: 4595 bytes --]