Project out of sources compilation

Project out of sources compilation

[Ergus]

Hi:

These days I have been working with cmake projects here and there and in
spite of I could handle most of the work from emacs, it required a lot
of extra time to make it more or less comfortable.

I wrote a couple of simple functions to manage my needs, but at the end
I think that there are some small piece missing to handle common
workflows and glue everything (with the users in mind of course):

1. Out of sources compilation.

Most of projects now prefer to do out-of sources compilation. Either to
keep source code clean or to keep multiple compilations at the same time
(i.e Debug/Release/win32)

The project.el package has already some compilation commands, but they
assume that the compilation will be executed in the project's
root... which is not true most of the time.

Maybe we may add an extra custom variable that could be specified in the
dir-locals.el in order specify where the compilation command must be
executed.

Some more heuristics here is possible, but I would settle for at least
something simpler.

2. Eglot compilations database place.

When compilation is out of sources the cmake generated
compile_commands.json also goes in that directory by default.

This issue can be managed with a line in dir-locals, or just manually
coping the database.

((eglot-workspace-configuration
	  . (:clangd (:initializationOptions (:compilationDatabasePath
	  "build")))))

Probably some simple slight integration of Eglot with Project may help.

   2.1 This mixes with the previous one because if we change the
   compilation directory the line with initializationOptions is not
   updated and requires manual intervention

3. Projects multiple backends

This one is tricky because at the moment I have gtags-mode, that
includes a backend for project.el, there is the default backend, but
also we could add something called cmake-backend (i.e looks for
CMakeLists.txt that includes a 'project' entry)

In that case emacs cannot use all the information form the three even if
it is not contradictory, so the user ends up opening the terminal and
doing things manually.

4. Flymake integration

Even without eglot, flymake should be capable to work very easily with
cmake projects.

This step is also a stage before doing a proper plugin integration of
tools like cppcheck for flymake.

5. Project local variables (a man can dream, a man can dream)

There are some situations where we want to have variables shared among a
project. (i.e some output directory, logging option when executing,
flags, environment variables).

At the moment these options work partially by using directory
variables. If we have the concept of a "project", maybe it is logical to
have some sort of project scope concept, specially for projects sharing
a common root.

For example vs-code adds a subdirectory with project variables that the
user (but also any plugin) can refer to in the project's scope.

--

I could try to implement some of this with your help; but I need some
feedback on which of them are desired and which are not. Or which ones
are maybe better to put as feature requests for a more skilled lisper or
package maintainer.

WDYT?
Ergus

Re: Project out of sources compilation

[Konstantin Kharlamov]

On Sat, 2024-03-16 at 14:12 +0100, Ergus wrote:
> 2. Eglot compilations database place.
> 
> When compilation is out of sources the cmake generated
> compile_commands.json also goes in that directory by default.
> 
> This issue can be managed with a line in dir-locals, or just manually
> coping the database.
> 
> ((eglot-workspace-configuration
> 	  . (:clangd (:initializationOptions
> (:compilationDatabasePath
> 	  "build")))))

Just wanted to clarify two things:

1. You can't just copy clangd database (i.e. `compile_commands.json`)
because the field "directory" has absolute path and clangd doesn't
support relative one. So you'll also have to regexp-replace the paths
after it's been copied.
2. If your out-of-source dir is called `build/`, then you don't need to
configure clangd to support it, because it handles that special case.

> Probably some simple slight integration of Eglot with Project may
> help.
> 
>    2.1 This mixes with the previous one because if we change the
>    compilation directory the line with initializationOptions is not
>    updated and requires manual intervention

Some generic support for pointing out the compilation dir sounds indeed
useful if it's possible.

Re: Project out of sources compilation

[Ergus]

Hi:

On Sat, Mar 16, 2024 at 07:50:49PM +0300, Konstantin Kharlamov wrote:
>On Sat, 2024-03-16 at 14:12 +0100, Ergus wrote:
>> 2. Eglot compilations database place.
>>
>> When compilation is out of sources the cmake generated
>> compile_commands.json also goes in that directory by default.
>>
>> This issue can be managed with a line in dir-locals, or just manually
>> coping the database.
>>
>> ((eglot-workspace-configuration
>> 	� . (:clangd (:initializationOptions
>> (:compilationDatabasePath
>> 	� "build")))))
>
>Just wanted to clarify two things:
>
>1. You can't just copy clangd database (i.e. `compile_commands.json`)
>because the field "directory" has absolute path and clangd doesn't
>support relative one. So you'll also have to regexp-replace the paths
>after it's been copied.

Yes, I know that (now). But that is actually the most popular solution in:

https://emacs.stackexchange.com/questions/73922/how-do-you-tell-eglot-and-clangd-about-the-compilation-database-compile-command

>2. If your out-of-source dir is called `build/`, then you don't need to
>configure clangd to support it, because it handles that special case.
>
Good to know, I didn't know because I always use names like build_debug,
build_x64, build_clang and so on.

>> Probably some simple slight integration of Eglot with Project may
>> help.
>>
>> �� 2.1 This mixes with the previous one because if we change the
>> �� compilation directory the line with initializationOptions is not
>> �� updated and requires manual intervention
>
>Some generic support for pointing out the compilation dir sounds indeed
>useful if it's possible.
>
Lets wait for Dmitry's suggestion.

Re: Project out of sources compilation

[Konstantin Kharlamov]

On Sat, 2024-03-16 at 20:00 +0100, Ergus wrote:
> Hi:
> 
> On Sat, Mar 16, 2024 at 07:50:49PM +0300, Konstantin Kharlamov wrote:
> > On Sat, 2024-03-16 at 14:12 +0100, Ergus wrote:
> > > 2. Eglot compilations database place.
> > > 
> > > When compilation is out of sources the cmake generated
> > > compile_commands.json also goes in that directory by default.
> > > 
> > > This issue can be managed with a line in dir-locals, or just
> > > manually
> > > coping the database.
> > > 
> > > ((eglot-workspace-configuration
> > > 	� . (:clangd (:initializationOptions
> > > (:compilationDatabasePath
> > > 	� "build")))))
> > 
> > Just wanted to clarify two things:
> > 
> > 1. You can't just copy clangd database (i.e.
> > `compile_commands.json`)
> > because the field "directory" has absolute path and clangd doesn't
> > support relative one. So you'll also have to regexp-replace the
> > paths
> > after it's been copied.
> 
> Yes, I know that (now). But that is actually the most popular
> solution in:
> 
> https://emacs.stackexchange.com/questions/73922/how-do-you-tell-eglot-and-clangd-about-the-compilation-database-compile-command

Actually, I stand corrected. I certainly remember that way not working
≈2 years ago (last time I tried and learned all that information);
however right now creating a symlink to `compile_commands.json` as well
as just copying does work.

I just tested that on a C project at work by symlinking/copying the
json file, then starting lsp and trying to "goto definition" to a
generated `.h` file that resides in the build dir. It works. I tested
that both with dir called `build` and `build-docs` (in the latter case
`build` dir was absent to make sure).

clangd version 16.0.6

Re: Project out of sources compilation

[Dmitry Gutov]

Hi!

On 16/03/2024 15:12, Ergus wrote:

> These days I have been working with cmake projects here and there and in
> spite of I could handle most of the work from emacs, it required a lot
> of extra time to make it more or less comfortable.
> 
> I wrote a couple of simple functions to manage my needs, but at the end
> I think that there are some small piece missing to handle common
> workflows and glue everything (with the users in mind of course):
> 
> 1. Out of sources compilation.
> 
> Most of projects now prefer to do out-of sources compilation. Either to
> keep source code clean or to keep multiple compilations at the same time
> (i.e Debug/Release/win32)
> 
> The project.el package has already some compilation commands, but they
> assume that the compilation will be executed in the project's
> root... which is not true most of the time.
> 
> Maybe we may add an extra custom variable that could be specified in the
> dir-locals.el in order specify where the compilation command must be
> executed.
> 
> Some more heuristics here is possible, but I would settle for at least
> something simpler.

project.el has just one, very simple command, where the only thing that 
it does it switch to the root first. How will you customize it? With a 
hook, where the user would write a function "determine a directory for 
compilation"? They might as well define a new command - or redefine this 
one. Or just an option with relative directory name?

> 2. Eglot compilations database place.
> 
> When compilation is out of sources the cmake generated
> compile_commands.json also goes in that directory by default.
> 
> This issue can be managed with a line in dir-locals, or just manually
> coping the database.
> 
> ((eglot-workspace-configuration
>        . (:clangd (:initializationOptions (:compilationDatabasePath
>        "build")))))
> 
> Probably some simple slight integration of Eglot with Project may help.

That seems like it can be a Eglot user option. Again, just storing a 
file name relative to the project root?

>    2.1 This mixes with the previous one because if we change the
>    compilation directory the line with initializationOptions is not
>    updated and requires manual intervention
> 
> 3. Projects multiple backends
> 
> This one is tricky because at the moment I have gtags-mode, that
> includes a backend for project.el, there is the default backend, but
> also we could add something called cmake-backend (i.e looks for
> CMakeLists.txt that includes a 'project' entry)
> 
> In that case emacs cannot use all the information form the three even if
> it is not contradictory, so the user ends up opening the terminal and
> doing things manually.

Indeed, project.el doesn't "merge" backends. Is there an obvious 
algorithm for merging backends which would be understandable to an 
average user?

If the only thing you wanted combined is the root detection, I suggest 
reusing the variable project-vc-extra-root-markers, automatically or 
manually (just have the user set it). If its data structure is now 
powerful enough, it can be updated. For example, if the logic both looks 
for CMakeLists.txt and checks its contents, 
project-vc-extra-root-markers could support cons entries like (MARKER . 
CHECK-FN).

> 4. Flymake integration
> 
> Even without eglot, flymake should be capable to work very easily with
> cmake projects.
> 
> This step is also a stage before doing a proper plugin integration of
> tools like cppcheck for flymake.

Is there something particular that is required?

> 5. Project local variables (a man can dream, a man can dream)
> 
> There are some situations where we want to have variables shared among a
> project. (i.e some output directory, logging option when executing,
> flags, environment variables).
> 
> At the moment these options work partially by using directory
> variables. If we have the concept of a "project", maybe it is logical to
> have some sort of project scope concept, specially for projects sharing
> a common root.
> 
> For example vs-code adds a subdirectory with project variables that the
> user (but also any plugin) can refer to in the project's scope.

Does such subdirectory in VS Code affect the project residing its its 
parent directory, or some other projects as well? Your description above 
sounds more like our .dir-locals.el.

And the directory locals facility in Emacs also has the less well-known 
capability called "directory class", see the bottom of 
https://www.gnu.org/software/emacs/manual/html_node/emacs/Directory-Variables.html. 
I think it should work okay for sharing variables between projects.

Perhaps it's less user-friendly than one had hoped, I don't know. Some 
improvements for related UIs would be welcome.

Just creating a parallel built-in way to set variables inside 
directories but for projects sounds a bit much. Also, we don't always 
detect a buffer's project before the user asks for it. So I'm not sure 
what hooks could be added, even if we wanted.

> I could try to implement some of this with your help; but I need some
> feedback on which of them are desired and which are not. Or which ones
> are maybe better to put as feature requests for a more skilled lisper or
> package maintainer.

Patches are welcome, but see above.

Cheers,
Dmitry.

Re: Project out of sources compilation

[Ergus]

Hi:

On March 17, 2024 3:53:42 AM GMT+01:00, Dmitry Gutov <dmitry@gutov.dev> wrote:
>Hi!
>
>On 16/03/2024 15:12, Ergus wrote:
>
>> These days I have been working with cmake projects here and there and in
>> spite of I could handle most of the work from emacs, it required a lot
>> of extra time to make it more or less comfortable.
>> 
>> I wrote a couple of simple functions to manage my needs, but at the end
>> I think that there are some small piece missing to handle common
>> workflows and glue everything (with the users in mind of course):
>> 
>> 1. Out of sources compilation.
>> 
>> Most of projects now prefer to do out-of sources compilation. Either to
>> keep source code clean or to keep multiple compilations at the same time
>> (i.e Debug/Release/win32)
>> 
>> The project.el package has already some compilation commands, but they
>> assume that the compilation will be executed in the project's
>> root... which is not true most of the time.
>> 
>> Maybe we may add an extra custom variable that could be specified in the
>> dir-locals.el in order specify where the compilation command must be
>> executed.
>> 
>> Some more heuristics here is possible, but I would settle for at least
>> something simpler.
>
>project.el has just one, very simple command, where the only thing that it does it switch to the root first. How will you customize it? With a hook, where the user would write a function "determine a directory for compilation"? They might as well define a new command - or redefine this one. Or just an option with relative directory name?
>

IMO the only thing we need is probably a variable/custom like project-build-dir. The user can define it in the dir-locals and the project command will use it if defined/ else use project-root. Maybe the backend could initialize it.

Alternatively (and not totally exclusive) project.el could define a project-build-dir function that project backends could optionally redefine (i.e I use a plist as project id in gags-mode and getting any stored property from there is very easy with a command). By default it will be an alias for project-root in the VC backend.

There is also some need for a 'bin' dir, that is, where the final executable will reside, useful to execute and debug with tools like gud and independent from 'build'...  For example in a python project this may be the project root OR where the file with __main__ resides, but a python project usually won't specify a build dir. But let's go for one thing at a time.


>> 2. Eglot compilations database place.
>> 
>> When compilation is out of sources the cmake generated
>> compile_commands.json also goes in that directory by default.
>> 
>> This issue can be managed with a line in dir-locals, or just manually
>> coping the database.
>> 
>> ((eglot-workspace-configuration
>>        . (:clangd (:initializationOptions (:compilationDatabasePath
>>        "build")))))
>> 
>> Probably some simple slight integration of Eglot with Project may help.
>
>That seems like it can be a Eglot user option. Again, just storing a file name relative to the project root?
>
Actually it doesn't need to be relative. For example QtCreator creates the build dir outside the sources directory in the same level. So, the only requirement is to be a valid path. Relative to root or absolute one

>>    2.1 This mixes with the previous one because if we change the
>>    compilation directory the line with initializationOptions is not
>>    updated and requires manual intervention
>> 
>> 3. Projects multiple backends
>> 
>> This one is tricky because at the moment I have gtags-mode, that
>> includes a backend for project.el, there is the default backend, but
>> also we could add something called cmake-backend (i.e looks for
>> CMakeLists.txt that includes a 'project' entry)
>> 
>> In that case emacs cannot use all the information form the three even if
>> it is not contradictory, so the user ends up opening the terminal and
>> doing things manually.
>
>Indeed, project.el doesn't "merge" backends. Is there an obvious algorithm for merging backends which would be understandable to an average user?
>
>If the only thing you wanted combined is the root detection, I suggest reusing the variable project-vc-extra-root-markers, automatically or manually (just have the user set it). If its data structure is now powerful enough, it can be updated. For example, if the logic both looks for CMakeLists.txt and checks its contents, project-vc-extra-root-markers could support cons entries like (MARKER . CHECK-FN).

Actually the idea is as simple as: if any of the backends explicitly specifies some property that the others don't, then use that one.

Applied in this case. VC backend does not have any "build" Information in general. But a cmake backend could detect a "build" directory and propose it (looking for a CMake cache inside it for example). An hipothetical  autotools backend may do the same.

>
>> 4. Flymake integration
>> 
>> Even without eglot, flymake should be capable to work very easily with
>> cmake projects.
>> 
>> This step is also a stage before doing a proper plugin integration of
>> tools like cppcheck for flymake.
>
>Is there something particular that is required?
>
>> 5. Project local variables (a man can dream, a man can dream)
>> 
>> There are some situations where we want to have variables shared among a
>> project. (i.e some output directory, logging option when executing,
>> flags, environment variables).
>> 
>> At the moment these options work partially by using directory
>> variables. If we have the concept of a "project", maybe it is logical to
>> have some sort of project scope concept, specially for projects sharing
>> a common root.
>> 
>> For example vs-code adds a subdirectory with project variables that the
>> user (but also any plugin) can refer to in the project's scope.
>
>Does such subdirectory in VS Code affect the project residing its its parent directory, or some other projects as well? Your description above sounds more like our .dir-locals.el.
>
>And the directory locals facility in Emacs also has the less well-known capability called "directory class", see the bottom of https://www.gnu.org/software/emacs/manual/html_node/emacs/Directory-Variables.html. I think it should work okay for sharing variables between projects.
>
>Perhaps it's less user-friendly than one had hoped, I don't know. Some improvements for related UIs would be welcome.
>
>Just creating a parallel built-in way to set variables inside directories but for projects sounds a bit much. Also, we don't always detect a buffer's project before the user asks for it. So I'm not sure what hooks could be added, even if we wanted.
>
Ok, I buy this one because there are the dir-locals, which are not ideal for project purposes, but it is already there and it isn't worth adding another abstraction layer

>> I could try to implement some of this with your help; but I need some
>> feedback on which of them are desired and which are not. Or which ones
>> are maybe better to put as feature requests for a more skilled lisper or
>> package maintainer.
>
>Patches are welcome, but see above.
>
>Cheers,
>Dmitry.
>

Best,
Ergus

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: Project out of sources compilation

[Eli Zaretskii]

> Date: Sun, 17 Mar 2024 08:22:56 +0100
> From: Ergus <spacibba@aol.com>
> 
> >> 1. Out of sources compilation.
> >> 
> >> Most of projects now prefer to do out-of sources compilation. Either to
> >> keep source code clean or to keep multiple compilations at the same time
> >> (i.e Debug/Release/win32)
> >> 
> >> The project.el package has already some compilation commands, but they
> >> assume that the compilation will be executed in the project's
> >> root... which is not true most of the time.
> >> 
> >> Maybe we may add an extra custom variable that could be specified in the
> >> dir-locals.el in order specify where the compilation command must be
> >> executed.
> >> 
> >> Some more heuristics here is possible, but I would settle for at least
> >> something simpler.
> >
> >project.el has just one, very simple command, where the only thing that it does it switch to the root first. How will you customize it? With a hook, where the user would write a function "determine a directory for compilation"? They might as well define a new command - or redefine this one. Or just an option with relative directory name?
> >
> 
> IMO the only thing we need is probably a variable/custom like project-build-dir. The user can define it in the dir-locals and the project command will use it if defined/ else use project-root. Maybe the backend could initialize it.
> 
> Alternatively (and not totally exclusive) project.el could define a project-build-dir function that project backends could optionally redefine (i.e I use a plist as project id in gags-mode and getting any stored property from there is very easy with a command). By default it will be an alias for project-root in the VC backend.

Maybe I'm missing something, but isn't the build tree just one more
tree that is part of the project?  If so, can't you use
project-external-roots to add this tree to the project?  I thought
this was the mechanism to add trees to a project as included in the
original design of project.el and its support in Emacs?

> There is also some need for a 'bin' dir, that is, where the final executable will reside, useful to execute and debug with tools like gud and independent from 'build'...  For example in a python project this may be the project root OR where the file with __main__ resides, but a python project usually won't specify a build dir. But let's go for one thing at a time.

Likewise here.

Or what am I missing?

Re: Project out of sources compilation

[Ergus]

On Sun, Mar 17, 2024 at 10:45:29AM +0200, Eli Zaretskii wrote:
>> Date: Sun, 17 Mar 2024 08:22:56 +0100
>> From: Ergus <spacibba@aol.com>
>>
>> >
>> >project.el has just one, very simple command, where the only thing that it does it switch to the root first. How will you customize it? With a hook, where the user would write a function "determine a directory for compilation"? They might as well define a new command - or redefine this one. Or just an option with relative directory name?
>> >
>>
>> IMO the only thing we need is probably a variable/custom like project-build-dir. The user can define it in the dir-locals and the project command will use it if defined/ else use project-root. Maybe the backend could initialize it.
>>
>> Alternatively (and not totally exclusive) project.el could define a project-build-dir function that project backends could optionally redefine (i.e I use a plist as project id in gags-mode and getting any stored property from there is very easy with a command). By default it will be an alias for project-root in the VC backend.
>
>Maybe I'm missing something, but isn't the build tree just one more
>tree that is part of the project?  If so, can't you use
>project-external-roots to add this tree to the project?  I thought
>this was the mechanism to add trees to a project as included in the
>original design of project.el and its support in Emacs?
>

>> There is also some need for a 'bin' dir, that is, where the final executable will reside, useful to execute and debug with tools like gud and independent from 'build'...  For example in a python project this may be the project root OR where the file with __main__ resides, but a python project usually won't specify a build dir. But let's go for one thing at a time.
>
>Likewise here.
>
>Or what am I missing?

Hi Eli:

More or less we have it, that's why I said that there was just some
(small) missing pieces.

We have the `project-external-roots`, but project.el uses them only to
find files and regexps. So it looks like they are intended to be source
places somehow.

The compile or debug programs doesn't know that they are intended to
execute there.

However, a `build` or `bin` are different because they are where the
`compile` or `gud` are intended to run; generally not a place to search
for sources (unless the source is generated like config.h).

Two examples:

```
git clone emacs
cd emacs
mkdir build # or mybuild, build_debug.. any name
./autogen.sh # (this is intendedto happen in the root)

cd build
../configure options  // This creates a Makefile
make

make install
```

On cmake

```
git clone project
cd project

   // Creates a CMakeCache.txt
   Alternative 1
     cmake -DCMAKE_EXPORT_COMPILE_COMMANDS=true -DCMAKE_BUILD_TYPE=Debug -B build/
     cd build

   Alternative 2
     mkdir build
     cd build
     cmake -DCMAKE_EXPORT_COMPILE_COMMANDS=true -DCMAKE_BUILD_TYPE=Debug ..

make
make install
```

The pattern is pretty much the same, so it is pretty simple to allow
projects.el to simplify the generation-compilation-execution-debug
workflow. And with autotools+cmake we are probably covering a huge range
of projects around.

Best,
Ergus

Re: Project out of sources compilation

[Eli Zaretskii]

> Date: Sun, 17 Mar 2024 18:33:21 +0100
> From: Ergus <spacibba@aol.com>
> Cc: dmitry@gutov.dev, emacs-devel@gnu.org
> 
> >Maybe I'm missing something, but isn't the build tree just one more
> >tree that is part of the project?  If so, can't you use
> >project-external-roots to add this tree to the project?  I thought
> >this was the mechanism to add trees to a project as included in the
> >original design of project.el and its support in Emacs?
> >
> 
> >> There is also some need for a 'bin' dir, that is, where the final executable will reside, useful to execute and debug with tools like gud and independent from 'build'...  For example in a python project this may be the project root OR where the file with __main__ resides, but a python project usually won't specify a build dir. But let's go for one thing at a time.
> >
> >Likewise here.
> >
> >Or what am I missing?
> 
> Hi Eli:
> 
> More or less we have it, that's why I said that there was just some
> (small) missing pieces.
> 
> We have the `project-external-roots`, but project.el uses them only to
> find files and regexps. So it looks like they are intended to be source
> places somehow.
> 
> The compile or debug programs doesn't know that they are intended to
> execute there.
> 
> However, a `build` or `bin` are different because they are where the
> `compile` or `gud` are intended to run; generally not a place to search
> for sources (unless the source is generated like config.h).

Are you saying that "M-x compile" and "M-x gdb" are not (yet)
supported by project.el?

Re: Project out of sources compilation

[Ergus]

On Sun, Mar 17, 2024 at 07:38:16PM +0200, Eli Zaretskii wrote:
>> Date: Sun, 17 Mar 2024 18:33:21 +0100
>> From: Ergus <spacibba@aol.com>
>> Cc: dmitry@gutov.dev, emacs-devel@gnu.org
>>
>> >Maybe I'm missing something, but isn't the build tree just one more
>> >tree that is part of the project?  If so, can't you use
>> >project-external-roots to add this tree to the project?  I thought
>> >this was the mechanism to add trees to a project as included in the
>> >original design of project.el and its support in Emacs?
>> >
>>
>> >> There is also some need for a 'bin' dir, that is, where the final executable will reside, useful to execute and debug with tools like gud and independent from 'build'...  For example in a python project this may be the project root OR where the file with __main__ resides, but a python project usually won't specify a build dir. But let's go for one thing at a time.
>> >
>> >Likewise here.
>> >
>> >Or what am I missing?
>>
>> Hi Eli:
>>
>> More or less we have it, that's why I said that there was just some
>> (small) missing pieces.
>>
>> We have the `project-external-roots`, but project.el uses them only to
>> find files and regexps. So it looks like they are intended to be source
>> places somehow.
>>
>> The compile or debug programs doesn't know that they are intended to
>> execute there.
>>
>> However, a `build` or `bin` are different because they are where the
>> `compile` or `gud` are intended to run; generally not a place to search
>> for sources (unless the source is generated like config.h).
>
>Are you saying that "M-x compile" and "M-x gdb" are not (yet)
>supported by project.el?
>

Kind of. There is project-compile, but executes in the root
directory. And M-x compile does not have any custom to control the
execution place, so it needs to do a let around it.

something similar happen to gdb, or project-eshell.

So, what is not actually supported is out of sources
compilation/execution. In the first email I explained that part. Even
Eglot requires extra work to function properly.

Re: Project out of sources compilation

[Augusto Stoffel]

On Sat, 16 Mar 2024 at 14:12, Ergus wrote:

> 5. Project local variables (a man can dream, a man can dream)
>
> There are some situations where we want to have variables shared among a
> project. (i.e some output directory, logging option when executing,
> flags, environment variables).
>
> At the moment these options work partially by using directory
> variables. If we have the concept of a "project", maybe it is logical to
> have some sort of project scope concept, specially for projects sharing
> a common root.

For Emacs variables, why you think dir-local variables only works
“partially”?

As to environment variables, they are generally assumed to be global in
Emacs and it goes a bit against the grain to make them buffer-local /
project dependent.  Nonetheless, there are packages for that (one is
buffer-env, which I wrote) and they work fine (at least for me).

> For example vs-code adds a subdirectory with project variables that the
> user (but also any plugin) can refer to in the project's scope.

Is this meant to store editor configuration or environment variables?  I
guess turning VSCode configuration variables into Emacs alternatives
would be tricky.  But if this is about env vars and the directory
organization is sensible, then one could configure buffer-env to
interoperate.

Re: Project out of sources compilation

[Ergus]

On Sun, Mar 17, 2024 at 12:36:44PM +0100, Augusto Stoffel wrote:
>On Sat, 16 Mar 2024 at 14:12, Ergus wrote:
>
>> 5. Project local variables (a man can dream, a man can dream)
>>
>> There are some situations where we want to have variables shared among a
>> project. (i.e some output directory, logging option when executing,
>> flags, environment variables).
>>
>> At the moment these options work partially by using directory
>> variables. If we have the concept of a "project", maybe it is logical to
>> have some sort of project scope concept, specially for projects sharing
>> a common root.
>
>For Emacs variables, why you think dir-local variables only works
>“partially”?
>
>As to environment variables, they are generally assumed to be global in
>Emacs and it goes a bit against the grain to make them buffer-local /
>project dependent.  Nonetheless, there are packages for that (one is
>buffer-env, which I wrote) and they work fine (at least for me).
>
They work for one directory, but when the project includes multiple
roots (project-external-roots) the variables are somehow missing just
when jumping with xref to a definition.

However, we can ignore this as I mentioned before.

>> For example vs-code adds a subdirectory with project variables that the
>> user (but also any plugin) can refer to in the project's scope.
>
>Is this meant to store editor configuration or environment variables?  I
>guess turning VSCode configuration variables into Emacs alternatives
>would be tricky. 

No please. No interaction-importing from other editors will be
maintainable in the long path.

> But if this is about env vars and the directory
>organization is sensible, then one could configure buffer-env to
>interoperate.

This is it. The idea is to extend project.el in order to bring more
project-oriented features.

Lets say:

The user (or project.el backend) could specify the `build` directory and
the environment that needs to be set to execute the program from the
build or bin directory.

So with a simple modification of dir-locals.el the user may be capable
to run M-x compile (or project-compile), M-x gud-gdb... or being more
ambitious M-x project-generate (to call autogen.sh and ./configure or
cmake depending of the project-backend)

For sure we wont cover the 100% of the projects, but wuth automake and
autotools we will be very close (and in the future someone could add
other backends for it)

Re: Project out of sources compilation

[Ergus]

[-- Attachment #1: Type: text/plain, Size: 2725 bytes --]

Hi Dmitry:

Here I attach a very simple approach to add support for out of sources
compilation in project.el

This is just a POC, but to have some starting point to discuss.

Best,
Ergus



On Sun, Mar 17, 2024 at 06:47:43PM +0100, Ergus wrote:
>On Sun, Mar 17, 2024 at 12:36:44PM +0100, Augusto Stoffel wrote:
>>On Sat, 16 Mar 2024 at 14:12, Ergus wrote:
>>
>>>5. Project local variables (a man can dream, a man can dream)
>>>
>>>There are some situations where we want to have variables shared among a
>>>project. (i.e some output directory, logging option when executing,
>>>flags, environment variables).
>>>
>>>At the moment these options work partially by using directory
>>>variables. If we have the concept of a "project", maybe it is logical to
>>>have some sort of project scope concept, specially for projects sharing
>>>a common root.
>>
>>For Emacs variables, why you think dir-local variables only works
>>“partially”?
>>
>>As to environment variables, they are generally assumed to be global in
>>Emacs and it goes a bit against the grain to make them buffer-local /
>>project dependent.  Nonetheless, there are packages for that (one is
>>buffer-env, which I wrote) and they work fine (at least for me).
>>
>They work for one directory, but when the project includes multiple
>roots (project-external-roots) the variables are somehow missing just
>when jumping with xref to a definition.
>
>However, we can ignore this as I mentioned before.
>
>>>For example vs-code adds a subdirectory with project variables that the
>>>user (but also any plugin) can refer to in the project's scope.
>>
>>Is this meant to store editor configuration or environment variables?  I
>>guess turning VSCode configuration variables into Emacs alternatives
>>would be tricky.
>
>No please. No interaction-importing from other editors will be
>maintainable in the long path.
>
>>But if this is about env vars and the directory
>>organization is sensible, then one could configure buffer-env to
>>interoperate.
>
>This is it. The idea is to extend project.el in order to bring more
>project-oriented features.
>
>Lets say:
>
>The user (or project.el backend) could specify the `build` directory and
>the environment that needs to be set to execute the program from the
>build or bin directory.
>
>So with a simple modification of dir-locals.el the user may be capable
>to run M-x compile (or project-compile), M-x gud-gdb... or being more
>ambitious M-x project-generate (to call autogen.sh and ./configure or
>cmake depending of the project-backend)
>
>For sure we wont cover the 100% of the projects, but wuth automake and
>autotools we will be very close (and in the future someone could add
>other backends for it)
>
>
>
>
>

[-- Attachment #2: project-build-dir.patch --]
[-- Type: text/plain, Size: 2875 bytes --]

commit cfa10dfeac01ae25a7acd6857fae0b377b625779
Author: Jimmy Aguilar Mena <kratsbinovish@gmail.com>
Date:   Tue Mar 19 17:57:25 2024 +0100

    Add basic project-build-dir implementation
    
    This is just a dummy prototype to discuss about alternatives.

diff --git a/lisp/progmodes/project.el b/lisp/progmodes/project.el
index ac18aceadcf..0ca4ca85566 100644
--- a/lisp/progmodes/project.el
+++ b/lisp/progmodes/project.el
@@ -51,6 +51,9 @@
 ;; files inside the root must not be considered a part of it).  It
 ;; should be consistent with `project-files'.
 ;;
+;; `project-build-dir' can be overridden if the project backend has some
+;; extra information about the project build directory.
+;;
 ;; This list can change in future versions.
 ;;
 ;; Transient project:
@@ -208,6 +211,12 @@ project-prompter
   :group 'project
   :version "30.1")
 
+(defcustom project-build-dir nil
+  "Build directory for current project."
+  :type 'directory
+  :safe t
+  :version "30.1")
+
 ;;;###autoload
 (defun project-current (&optional maybe-prompt directory)
   "Return the project instance in DIRECTORY, defaulting to `default-directory'.
@@ -289,8 +298,30 @@ project-external-roots
 headers search path, load path, class path, and so on."
   nil)
 
+(cl-defgeneric project-build-dir (_project)
+  "Return build directory of the current PROJECT.
+
+This function is intended to be defined by the backend when possible.
+Otherwise this returns nil and the `project-compile' command will be
+called in the project-root.
+If the user defines the directory-local variable `project-build-dir' it
+will have precedence over the result of this function."
+ nil)
+
+(defun project-get-build-dir (project)
+  "Return build directory of the current PROJECT.
+If the variable `project-build-dir' is defined, this function returns
+it, otherwise it returns the project root.  If the defined path is
+relative, this expands it relatively to the project's root"
+  (let ((dir (or project-build-dir
+                 (project-build-dir project)
+                 (project-root project)))) ;; I assume project-root is absolute
+    (if (file-name-absolute-p dir)
+        dir
+      (expand-file-name dir (project-root project)))))
+
 (cl-defgeneric project-name (project)
-  "A human-readable name for the project.
+  "A human-readable name for the PROJECT.
 Nominally unique, but not enforced."
   (file-name-nondirectory (directory-file-name (project-root project))))
 
@@ -1391,7 +1422,7 @@ project-compile
   "Run `compile' in the project root."
   (declare (interactive-only compile))
   (interactive)
-  (let ((default-directory (project-root (project-current t)))
+  (let ((default-directory (project-get-build-dir (project-current t)))
         (compilation-buffer-name-function
          (or project-compilation-buffer-name-function
              compilation-buffer-name-function)))

[PATCH] Project out of sources compilation

[Ergus]

[-- Attachment #1: Type: text/plain, Size: 6361 bytes --]

Hi:

Here I attach a very simple patch to allow project out of sources
compilation and project build detection.

So far I already implemented a very primitive backend [1] for project.el
using these features to detect and compile autotools and cmake projects;
and, at some point I will propose it for elpa.

But I need to work a bit more on it; and, of course, the functionalities
in the attached patch need to go to vanilla this way or some other
equivalent.

Best,
Ergus

[1] https://github.com/Ergus/project-multi-mode/blob/master/project-multi-mode.el



On Tue, Mar 19, 2024 at 07:36:35PM +0100, Ergus wrote:
>Hi Dmitry:
>
>Here I attach a very simple approach to add support for out of sources
>compilation in project.el
>
>This is just a POC, but to have some starting point to discuss.
>
>Best,
>Ergus
>
>
>
>On Sun, Mar 17, 2024 at 06:47:43PM +0100, Ergus wrote:
>>On Sun, Mar 17, 2024 at 12:36:44PM +0100, Augusto Stoffel wrote:
>>>On Sat, 16 Mar 2024 at 14:12, Ergus wrote:
>>>
>>>>5. Project local variables (a man can dream, a man can dream)
>>>>
>>>>There are some situations where we want to have variables shared among a
>>>>project. (i.e some output directory, logging option when executing,
>>>>flags, environment variables).
>>>>
>>>>At the moment these options work partially by using directory
>>>>variables. If we have the concept of a "project", maybe it is logical to
>>>>have some sort of project scope concept, specially for projects sharing
>>>>a common root.
>>>
>>>For Emacs variables, why you think dir-local variables only works
>>>“partially”?
>>>
>>>As to environment variables, they are generally assumed to be global in
>>>Emacs and it goes a bit against the grain to make them buffer-local /
>>>project dependent.  Nonetheless, there are packages for that (one is
>>>buffer-env, which I wrote) and they work fine (at least for me).
>>>
>>They work for one directory, but when the project includes multiple
>>roots (project-external-roots) the variables are somehow missing just
>>when jumping with xref to a definition.
>>
>>However, we can ignore this as I mentioned before.
>>
>>>>For example vs-code adds a subdirectory with project variables that the
>>>>user (but also any plugin) can refer to in the project's scope.
>>>
>>>Is this meant to store editor configuration or environment variables?  I
>>>guess turning VSCode configuration variables into Emacs alternatives
>>>would be tricky.
>>
>>No please. No interaction-importing from other editors will be
>>maintainable in the long path.
>>
>>>But if this is about env vars and the directory
>>>organization is sensible, then one could configure buffer-env to
>>>interoperate.
>>
>>This is it. The idea is to extend project.el in order to bring more
>>project-oriented features.
>>
>>Lets say:
>>
>>The user (or project.el backend) could specify the `build` directory and
>>the environment that needs to be set to execute the program from the
>>build or bin directory.
>>
>>So with a simple modification of dir-locals.el the user may be capable
>>to run M-x compile (or project-compile), M-x gud-gdb... or being more
>>ambitious M-x project-generate (to call autogen.sh and ./configure or
>>cmake depending of the project-backend)
>>
>>For sure we wont cover the 100% of the projects, but wuth automake and
>>autotools we will be very close (and in the future someone could add
>>other backends for it)
>>
>>
>>
>>
>>

>commit cfa10dfeac01ae25a7acd6857fae0b377b625779
>Author: Jimmy Aguilar Mena <kratsbinovish@gmail.com>
>Date:   Tue Mar 19 17:57:25 2024 +0100
>
>    Add basic project-build-dir implementation
>
>    This is just a dummy prototype to discuss about alternatives.
>
>diff --git a/lisp/progmodes/project.el b/lisp/progmodes/project.el
>index ac18aceadcf..0ca4ca85566 100644
>--- a/lisp/progmodes/project.el
>+++ b/lisp/progmodes/project.el
>@@ -51,6 +51,9 @@
> ;; files inside the root must not be considered a part of it).  It
> ;; should be consistent with `project-files'.
> ;;
>+;; `project-build-dir' can be overridden if the project backend has some
>+;; extra information about the project build directory.
>+;;
> ;; This list can change in future versions.
> ;;
> ;; Transient project:
>@@ -208,6 +211,12 @@ project-prompter
>   :group 'project
>   :version "30.1")
>
>+(defcustom project-build-dir nil
>+  "Build directory for current project."
>+  :type 'directory
>+  :safe t
>+  :version "30.1")
>+
> ;;;###autoload
> (defun project-current (&optional maybe-prompt directory)
>   "Return the project instance in DIRECTORY, defaulting to `default-directory'.
>@@ -289,8 +298,30 @@ project-external-roots
> headers search path, load path, class path, and so on."
>   nil)
>
>+(cl-defgeneric project-build-dir (_project)
>+  "Return build directory of the current PROJECT.
>+
>+This function is intended to be defined by the backend when possible.
>+Otherwise this returns nil and the `project-compile' command will be
>+called in the project-root.
>+If the user defines the directory-local variable `project-build-dir' it
>+will have precedence over the result of this function."
>+ nil)
>+
>+(defun project-get-build-dir (project)
>+  "Return build directory of the current PROJECT.
>+If the variable `project-build-dir' is defined, this function returns
>+it, otherwise it returns the project root.  If the defined path is
>+relative, this expands it relatively to the project's root"
>+  (let ((dir (or project-build-dir
>+                 (project-build-dir project)
>+                 (project-root project)))) ;; I assume project-root is absolute
>+    (if (file-name-absolute-p dir)
>+        dir
>+      (expand-file-name dir (project-root project)))))
>+
> (cl-defgeneric project-name (project)
>-  "A human-readable name for the project.
>+  "A human-readable name for the PROJECT.
> Nominally unique, but not enforced."
>   (file-name-nondirectory (directory-file-name (project-root project))))
>
>@@ -1391,7 +1422,7 @@ project-compile
>   "Run `compile' in the project root."
>   (declare (interactive-only compile))
>   (interactive)
>-  (let ((default-directory (project-root (project-current t)))
>+  (let ((default-directory (project-get-build-dir (project-current t)))
>         (compilation-buffer-name-function
>          (or project-compilation-buffer-name-function
>              compilation-buffer-name-function)))


[-- Attachment #2: project-compile.patch --]
[-- Type: text/plain, Size: 4160 bytes --]

diff --git a/lisp/progmodes/project.el b/lisp/progmodes/project.el
index ac18aceadcf..5a4274d6f68 100644
--- a/lisp/progmodes/project.el
+++ b/lisp/progmodes/project.el
@@ -51,6 +51,9 @@
 ;; files inside the root must not be considered a part of it).  It
 ;; should be consistent with `project-files'.
 ;;
+;; `project-build-dir' can be overridden if the project backend has some
+;; extra information about the project build directory.
+;;
 ;; This list can change in future versions.
 ;;
 ;; Transient project:
@@ -208,6 +211,22 @@ project-prompter
   :group 'project
   :version "30.1")
 
+(defcustom project-build-dir nil
+  "Build directory for current project.
+This is the custom that the user could specify in dir-locals to override
+the option specified by the project's backend."
+  :type 'directory
+  :safe t
+  :version "30.1")
+
+(defcustom project-compile-command nil
+  "Build command for current project.
+This is the custom that the user could specify in dir-locals to override
+the option specified by the project's backend."
+  :type 'directory
+  :safe t
+  :version "30.1")
+
 ;;;###autoload
 (defun project-current (&optional maybe-prompt directory)
   "Return the project instance in DIRECTORY, defaulting to `default-directory'.
@@ -289,8 +308,42 @@ project-external-roots
 headers search path, load path, class path, and so on."
   nil)
 
+(cl-defgeneric project-build-dir (_project)
+  "Return build directory of the current PROJECT.
+
+This function is intended to be defined by the backend when possible.
+Otherwise this returns nil and the `project-compile' command will be
+called in the project-root.
+If the user defines the directory-local variable `project-build-dir' it
+will have precedence over the result of this function."
+ nil)
+
+(defun project-get-build-dir (project)
+  "Return build directory of the current PROJECT.
+1. If the variable `project-build-dir' is defined, this function returns
+it.  2. Else if the function `project-build-dir' return non-nil.
+3. else it return the project-root.
+If the defined path is relative, this expands it relatively to the
+project's root."
+  (let ((dir (or project-build-dir           ;; variable for dir-locals or connection local vars
+                 (project-build-dir project) ;; backend function
+                 (project-root project))))   ;; I assume project-root is always absolute
+    (if (file-name-absolute-p dir)
+        dir
+      (expand-file-name dir (project-root project)))))
+
+(cl-defgeneric project-compile-command (_project)
+  "Return build command of the current PROJECT.
+
+This function is intended to be defined by the backend when possible.
+Otherwise this returns nil and the `project-compile' command will use
+the default `compile-command' value.  If the user defines the
+directory-local variable `project-build-command' it will have preference
+over this function and this will be never called."
+ nil)
+
 (cl-defgeneric project-name (project)
-  "A human-readable name for the project.
+  "A human-readable name for the PROJECT.
 Nominally unique, but not enforced."
   (file-name-nondirectory (directory-file-name (project-root project))))
 
@@ -1391,10 +1444,16 @@ project-compile
   "Run `compile' in the project root."
   (declare (interactive-only compile))
   (interactive)
-  (let ((default-directory (project-root (project-current t)))
-        (compilation-buffer-name-function
-         (or project-compilation-buffer-name-function
-             compilation-buffer-name-function)))
+  ;; I am wondering whenever we need to expand connection local
+  ;; variables at this point... maybe before or inside the let.
+  (let* ((project (project-current t))
+         (default-directory (project-get-build-dir project))
+         (compile-command (or project-compile-command
+                              (project-compile-command project)
+                              compile-command))
+         (compilation-buffer-name-function
+          (or project-compilation-buffer-name-function
+              compilation-buffer-name-function)))
     (call-interactively #'compile)))
 
 (defun project-recompile (&optional edit-command)

Re: [PATCH] Project out of sources compilation

[Dmitry Gutov]

Hi Ergus,

On 27/03/2024 18:38, Ergus wrote:

> Here I attach a very simple patch to allow project out of sources
> compilation and project build detection.
> 
> So far I already implemented a very primitive backend [1] for project.el
> using these features to detect and compile autotools and cmake projects;
> and, at some point I will propose it for elpa.
> 
> But I need to work a bit more on it; and, of course, the functionalities
> in the attached patch need to go to vanilla this way or some other
> equivalent.

I originally suggested a variable that stores a relative file name. But 
I see that you prefer the fetching of this file name to be more dynamic.

Why don't we just make it a function variable, at least at first?

Call it 'project-compile-dir`, which would be settable to a string 
(constant value, to be customized through dir-locals), or a function 
that either accepts the current project value, or is simply called 
"inside project root" (with default-directory bound to that value).

Alternatively, if you see a need to add additional methods 
('project-compile-command'?), it could become a new hook and a new 
facility (e.g. project-build-functions) which could be structures 
similarly to project.el, i.e. the hook returns a "project builder" 
value, and the value has some methods that it can defined/override. Then 
different build tool backends could be mixed with different project 
backends more easily.

So far I only see one additional method, though, and the constructed 
compile-command value is relatively simple. Curious to see whether this 
will get more complex with additional targets.

As far as the redefinition of 'project-name' goes, perhaps we'll allow 
project-vc-name to be a function as well? Or decide on some other way to 
assign a build tool the name to a project.

> Best,
> Ergus
> 
> [1] 
> https://github.com/Ergus/project-multi-mode/blob/master/project-multi-mode.el
> 
> 
> 
> On Tue, Mar 19, 2024 at 07:36:35PM +0100, Ergus wrote:
>> Hi Dmitry:
>>
>> Here I attach a very simple approach to add support for out of sources
>> compilation in project.el
>>
>> This is just a POC, but to have some starting point to discuss.
>>
>> Best,
>> Ergus
>>
>>
>>
>> On Sun, Mar 17, 2024 at 06:47:43PM +0100, Ergus wrote:
>>> On Sun, Mar 17, 2024 at 12:36:44PM +0100, Augusto Stoffel wrote:
>>>> On Sat, 16 Mar 2024 at 14:12, Ergus wrote:
>>>>
>>>>> 5. Project local variables (a man can dream, a man can dream)
>>>>>
>>>>> There are some situations where we want to have variables shared 
>>>>> among a
>>>>> project. (i.e some output directory, logging option when executing,
>>>>> flags, environment variables).
>>>>>
>>>>> At the moment these options work partially by using directory
>>>>> variables. If we have the concept of a "project", maybe it is 
>>>>> logical to
>>>>> have some sort of project scope concept, specially for projects 
>>>>> sharing
>>>>> a common root.
>>>>
>>>> For Emacs variables, why you think dir-local variables only works
>>>> “partially”?
>>>>
>>>> As to environment variables, they are generally assumed to be global in
>>>> Emacs and it goes a bit against the grain to make them buffer-local /
>>>> project dependent.  Nonetheless, there are packages for that (one is
>>>> buffer-env, which I wrote) and they work fine (at least for me).
>>>>
>>> They work for one directory, but when the project includes multiple
>>> roots (project-external-roots) the variables are somehow missing just
>>> when jumping with xref to a definition.
>>>
>>> However, we can ignore this as I mentioned before.
>>>
>>>>> For example vs-code adds a subdirectory with project variables that 
>>>>> the
>>>>> user (but also any plugin) can refer to in the project's scope.
>>>>
>>>> Is this meant to store editor configuration or environment 
>>>> variables?  I
>>>> guess turning VSCode configuration variables into Emacs alternatives
>>>> would be tricky.
>>>
>>> No please. No interaction-importing from other editors will be
>>> maintainable in the long path.
>>>
>>>> But if this is about env vars and the directory
>>>> organization is sensible, then one could configure buffer-env to
>>>> interoperate.
>>>
>>> This is it. The idea is to extend project.el in order to bring more
>>> project-oriented features.
>>>
>>> Lets say:
>>>
>>> The user (or project.el backend) could specify the `build` directory and
>>> the environment that needs to be set to execute the program from the
>>> build or bin directory.
>>>
>>> So with a simple modification of dir-locals.el the user may be capable
>>> to run M-x compile (or project-compile), M-x gud-gdb... or being more
>>> ambitious M-x project-generate (to call autogen.sh and ./configure or
>>> cmake depending of the project-backend)
>>>
>>> For sure we wont cover the 100% of the projects, but wuth automake and
>>> autotools we will be very close (and in the future someone could add
>>> other backends for it)
>>>
>>>
>>>
>>>
>>>
> 
>> commit cfa10dfeac01ae25a7acd6857fae0b377b625779
>> Author: Jimmy Aguilar Mena <kratsbinovish@gmail.com>
>> Date:   Tue Mar 19 17:57:25 2024 +0100
>>
>>    Add basic project-build-dir implementation
>>
>>    This is just a dummy prototype to discuss about alternatives.
>>
>> diff --git a/lisp/progmodes/project.el b/lisp/progmodes/project.el
>> index ac18aceadcf..0ca4ca85566 100644
>> --- a/lisp/progmodes/project.el
>> +++ b/lisp/progmodes/project.el
>> @@ -51,6 +51,9 @@
>> ;; files inside the root must not be considered a part of it).  It
>> ;; should be consistent with `project-files'.
>> ;;
>> +;; `project-build-dir' can be overridden if the project backend has some
>> +;; extra information about the project build directory.
>> +;;
>> ;; This list can change in future versions.
>> ;;
>> ;; Transient project:
>> @@ -208,6 +211,12 @@ project-prompter
>>   :group 'project
>>   :version "30.1")
>>
>> +(defcustom project-build-dir nil
>> +  "Build directory for current project."
>> +  :type 'directory
>> +  :safe t
>> +  :version "30.1")
>> +
>> ;;;###autoload
>> (defun project-current (&optional maybe-prompt directory)
>>   "Return the project instance in DIRECTORY, defaulting to 
>> `default-directory'.
>> @@ -289,8 +298,30 @@ project-external-roots
>> headers search path, load path, class path, and so on."
>>   nil)
>>
>> +(cl-defgeneric project-build-dir (_project)
>> +  "Return build directory of the current PROJECT.
>> +
>> +This function is intended to be defined by the backend when possible.
>> +Otherwise this returns nil and the `project-compile' command will be
>> +called in the project-root.
>> +If the user defines the directory-local variable `project-build-dir' it
>> +will have precedence over the result of this function."
>> + nil)
>> +
>> +(defun project-get-build-dir (project)
>> +  "Return build directory of the current PROJECT.
>> +If the variable `project-build-dir' is defined, this function returns
>> +it, otherwise it returns the project root.  If the defined path is
>> +relative, this expands it relatively to the project's root"
>> +  (let ((dir (or project-build-dir
>> +                 (project-build-dir project)
>> +                 (project-root project)))) ;; I assume project-root 
>> is absolute
>> +    (if (file-name-absolute-p dir)
>> +        dir
>> +      (expand-file-name dir (project-root project)))))
>> +
>> (cl-defgeneric project-name (project)
>> -  "A human-readable name for the project.
>> +  "A human-readable name for the PROJECT.
>> Nominally unique, but not enforced."
>>   (file-name-nondirectory (directory-file-name (project-root project))))
>>
>> @@ -1391,7 +1422,7 @@ project-compile
>>   "Run `compile' in the project root."
>>   (declare (interactive-only compile))
>>   (interactive)
>> -  (let ((default-directory (project-root (project-current t)))
>> +  (let ((default-directory (project-get-build-dir (project-current t)))
>>         (compilation-buffer-name-function
>>          (or project-compilation-buffer-name-function
>>              compilation-buffer-name-function)))
> 

Re: [PATCH] Project out of sources compilation

[Ergus]

Hi Dmitry:

On Sun, Mar 31, 2024 at 05:41:24AM +0300, Dmitry Gutov wrote:
>Hi Ergus,
>
>On 27/03/2024 18:38, Ergus wrote:
>
>>Here I attach a very simple patch to allow project out of sources
>>compilation and project build detection.
>>
>>So far I already implemented a very primitive backend [1] for project.el
>>using these features to detect and compile autotools and cmake projects;
>>and, at some point I will propose it for elpa.
>>
>>But I need to work a bit more on it; and, of course, the functionalities
>>in the attached patch need to go to vanilla this way or some other
>>equivalent.
>
>I originally suggested a variable that stores a relative file name. 
>But I see that you prefer the fetching of this file name to be more 
>dynamic.
>
Yes, that's actually the point. In out of sources compilation there is a
possibility that there will be more than one subdirectory to build, in
that case the user may be asked at least once which one to choose. This
is also valid to detect the `compile-commands.json' which may have
different version in each of the build-dir candidates.

https://github.com/Ergus/project-multi-mode/blob/c679d73a26b162e4158f6451027ca94e33faca0b/project-multi-mode.el#L109

>Why don't we just make it a function variable, at least at first?
>
Because I wanted somehow to follow the project.el schema of using a
generic that the backend could optionally define transparently.

And make that definition independent from the user config variable, so,
the user defined variable takes preference over the generic search
performed by the backend which if not defined automatically goes to
project root. Somehow adding a custom variable I think interferes with
the idea that the backend provides the functionality... But I may be
wrong.

>Call it 'project-compile-dir`, which would be settable to a string 
>(constant value, to be customized through dir-locals), or a function 
>that either accepts the current project value, or is simply called 
>"inside project root" (with default-directory bound to that value).
>
I will try this again, but I had some issues before.

>Alternatively, if you see a need to add additional methods 
>('project-compile-command'?), it could become a new hook and a new 
>facility (e.g. project-build-functions) which could be structures 
>similarly to project.el, i.e. the hook returns a "project builder" 
>value, and the value has some methods that it can defined/override. 
>Then different build tool backends could be mixed with different 
>project backends more easily.
>
I would prefer avoid more abstraction layers in the api. We actually
need some `project-compile-command' like function, but structuring it
with generics and more layer is IMHO adding too much complexity for a
public api... Isn't there a simpler alternative?

>So far I only see one additional method, though, and the constructed 
>compile-command value is relatively simple. Curious to see whether 
>this will get more complex with additional targets.
>
I added one method and with that I already support autotools and cmake.

>As far as the redefinition of 'project-name' goes, perhaps we'll allow 
>project-vc-name to be a function as well? Or decide on some other way 
>to assign a build tool the name to a project.
>
In project-name I actually only changed the docstring, I didn't change
anything else.

>>Best,
>>Ergus
>>
>>[1] https://github.com/Ergus/project-multi-mode/blob/master/project-multi-mode.el
>>
>>
>>
>>On Tue, Mar 19, 2024 at 07:36:35PM +0100, Ergus wrote:
>>>Hi Dmitry:
>>>
>>>Here I attach a very simple approach to add support for out of sources
>>>compilation in project.el
>>>
>>>This is just a POC, but to have some starting point to discuss.
>>>
>>>Best,
>>>Ergus
>>>
>>>
>>>
>>>On Sun, Mar 17, 2024 at 06:47:43PM +0100, Ergus wrote:
>>>>On Sun, Mar 17, 2024 at 12:36:44PM +0100, Augusto Stoffel wrote:
>>>>>On Sat, 16 Mar 2024 at 14:12, Ergus wrote:
>>>>>
>>>>>>5. Project local variables (a man can dream, a man can dream)
>>>>>>
>>>>>>There are some situations where we want to have variables 
>>>>>>shared among a
>>>>>>project. (i.e some output directory, logging option when executing,
>>>>>>flags, environment variables).
>>>>>>
>>>>>>At the moment these options work partially by using directory
>>>>>>variables. If we have the concept of a "project", maybe it 
>>>>>>is logical to
>>>>>>have some sort of project scope concept, specially for 
>>>>>>projects sharing
>>>>>>a common root.
>>>>>
>>>>>For Emacs variables, why you think dir-local variables only works
>>>>>“partially”?
>>>>>
>>>>>As to environment variables, they are generally assumed to be global in
>>>>>Emacs and it goes a bit against the grain to make them buffer-local /
>>>>>project dependent.  Nonetheless, there are packages for that (one is
>>>>>buffer-env, which I wrote) and they work fine (at least for me).
>>>>>
>>>>They work for one directory, but when the project includes multiple
>>>>roots (project-external-roots) the variables are somehow missing just
>>>>when jumping with xref to a definition.
>>>>
>>>>However, we can ignore this as I mentioned before.
>>>>
>>>>>>For example vs-code adds a subdirectory with project 
>>>>>>variables that the
>>>>>>user (but also any plugin) can refer to in the project's scope.
>>>>>
>>>>>Is this meant to store editor configuration or environment 
>>>>>variables?  I
>>>>>guess turning VSCode configuration variables into Emacs alternatives
>>>>>would be tricky.
>>>>
>>>>No please. No interaction-importing from other editors will be
>>>>maintainable in the long path.
>>>>
>>>>>But if this is about env vars and the directory
>>>>>organization is sensible, then one could configure buffer-env to
>>>>>interoperate.
>>>>
>>>>This is it. The idea is to extend project.el in order to bring more
>>>>project-oriented features.
>>>>
>>>>Lets say:
>>>>
>>>>The user (or project.el backend) could specify the `build` directory and
>>>>the environment that needs to be set to execute the program from the
>>>>build or bin directory.
>>>>
>>>>So with a simple modification of dir-locals.el the user may be capable
>>>>to run M-x compile (or project-compile), M-x gud-gdb... or being more
>>>>ambitious M-x project-generate (to call autogen.sh and ./configure or
>>>>cmake depending of the project-backend)
>>>>
>>>>For sure we wont cover the 100% of the projects, but wuth automake and
>>>>autotools we will be very close (and in the future someone could add
>>>>other backends for it)
>>>>
>>>>
>>>>
>>>>
>>>>
>>
>>>commit cfa10dfeac01ae25a7acd6857fae0b377b625779
>>>Author: Jimmy Aguilar Mena <kratsbinovish@gmail.com>
>>>Date:   Tue Mar 19 17:57:25 2024 +0100
>>>
>>>   Add basic project-build-dir implementation
>>>
>>>   This is just a dummy prototype to discuss about alternatives.
>>>
>>>diff --git a/lisp/progmodes/project.el b/lisp/progmodes/project.el
>>>index ac18aceadcf..0ca4ca85566 100644
>>>--- a/lisp/progmodes/project.el
>>>+++ b/lisp/progmodes/project.el
>>>@@ -51,6 +51,9 @@
>>>;; files inside the root must not be considered a part of it).  It
>>>;; should be consistent with `project-files'.
>>>;;
>>>+;; `project-build-dir' can be overridden if the project backend has some
>>>+;; extra information about the project build directory.
>>>+;;
>>>;; This list can change in future versions.
>>>;;
>>>;; Transient project:
>>>@@ -208,6 +211,12 @@ project-prompter
>>>  :group 'project
>>>  :version "30.1")
>>>
>>>+(defcustom project-build-dir nil
>>>+  "Build directory for current project."
>>>+  :type 'directory
>>>+  :safe t
>>>+  :version "30.1")
>>>+
>>>;;;###autoload
>>>(defun project-current (&optional maybe-prompt directory)
>>>  "Return the project instance in DIRECTORY, defaulting to 
>>>`default-directory'.
>>>@@ -289,8 +298,30 @@ project-external-roots
>>>headers search path, load path, class path, and so on."
>>>  nil)
>>>
>>>+(cl-defgeneric project-build-dir (_project)
>>>+  "Return build directory of the current PROJECT.
>>>+
>>>+This function is intended to be defined by the backend when possible.
>>>+Otherwise this returns nil and the `project-compile' command will be
>>>+called in the project-root.
>>>+If the user defines the directory-local variable `project-build-dir' it
>>>+will have precedence over the result of this function."
>>>+ nil)
>>>+
>>>+(defun project-get-build-dir (project)
>>>+  "Return build directory of the current PROJECT.
>>>+If the variable `project-build-dir' is defined, this function returns
>>>+it, otherwise it returns the project root.  If the defined path is
>>>+relative, this expands it relatively to the project's root"
>>>+  (let ((dir (or project-build-dir
>>>+                 (project-build-dir project)
>>>+                 (project-root project)))) ;; I assume 
>>>project-root is absolute
>>>+    (if (file-name-absolute-p dir)
>>>+        dir
>>>+      (expand-file-name dir (project-root project)))))
>>>+
>>>(cl-defgeneric project-name (project)
>>>-  "A human-readable name for the project.
>>>+  "A human-readable name for the PROJECT.
>>>Nominally unique, but not enforced."
>>>  (file-name-nondirectory (directory-file-name (project-root project))))
>>>
>>>@@ -1391,7 +1422,7 @@ project-compile
>>>  "Run `compile' in the project root."
>>>  (declare (interactive-only compile))
>>>  (interactive)
>>>-  (let ((default-directory (project-root (project-current t)))
>>>+  (let ((default-directory (project-get-build-dir (project-current t)))
>>>        (compilation-buffer-name-function
>>>         (or project-compilation-buffer-name-function
>>>             compilation-buffer-name-function)))
>>
>
>

Re: [PATCH] Project out of sources compilation

[Dirk-Jan C. Binnema]

On Sunday Mar 31 2024, Ergus wrote:

> Hi Dmitry:
>
> On Sun, Mar 31, 2024 at 05:41:24AM +0300, Dmitry Gutov wrote:
>>Hi Ergus,
>>
>>On 27/03/2024 18:38, Ergus wrote:
>>

>> I originally suggested a variable that stores a relative file name. But I see
>> that you prefer the fetching of this file name to be more dynamic.
>>
> Yes, that's actually the point. In out of sources compilation there is a
> possibility that there will be more than one subdirectory to build, in
> that case the user may be asked at least once which one to choose. This
> is also valid to detect the `compile-commands.json' which may have
> different version in each of the build-dir candidates.
>
> https://github.com/Ergus/project-multi-mode/blob/c679d73a26b162e4158f6451027ca94e33faca0b/project-multi-mode.el#L109

My 0.02: Having separate build-directories seems to be increasingly
common (in my bubble I hardly see anything else anymore)

Allowing for _multiple_ build-directories is also quit common/useful.
E.g., for doing debug/release builds, or for building for different
target hw (embedded dev). For me, a daily need.

So having direct support for this in project.el would be quite welcome,
so I can use that instead of my private hacks.

>>Why don't we just make it a function variable, at least at first?
>>
> Because I wanted somehow to follow the project.el schema of using a
> generic that the backend could optionally define transparently.
>
> And make that definition independent from the user config variable, so,
> the user defined variable takes preference over the generic search
> performed by the backend which if not defined automatically goes to
> project root. Somehow adding a custom variable I think interferes with
> the idea that the backend provides the functionality... But I may be
> wrong.
>
>> Call it 'project-compile-dir`, which would be settable to a string (constant
>> value, to be customized through dir-locals), or a function that either accepts
>> the current project value, or is simply called "inside project root" (with
>> default-directory bound to that value).
>>
> I will try this again, but I had some issues before.
>
>> Alternatively, if you see a need to add additional methods
>> ('project-compile-command'?), it could become a new hook and a new facility
>> (e.g. project-build-functions) which could be structures similarly to
>> project.el, i.e. the hook returns a "project builder" value, and the value has
>> some methods that it can defined/override. Then different build tool backends
>> could be mixed with different project backends more easily.
>>
> I would prefer avoid more abstraction layers in the api. We actually
> need some `project-compile-command' like function, but structuring it
> with generics and more layer is IMHO adding too much complexity for a
> public api... Isn't there a simpler alternative?
>
>> So far I only see one additional method, though, and the constructed
>> compile-command value is relatively simple. Curious to see whether this will
>> get more complex with additional targets.
>>
> I added one method and with that I already support autotools and cmake.

(would be nice to add "meson" too!)

No immediate opinions on how to implement this, but things I commonly
want in projects:

- build
- run
- test
- install
- flash
- debug
- clean

It's possible of course to "multiplex" with the compile-command and put
in a "transient" or something like that; but it'd be nice to handle this
directly across projects; also for adding menu / toolbar commands.

Guess this is a bit of an open-ended list, so having `project-*-command`
may not to be the best.

Kind regards,
Dirk.

-- 
Dirk-Jan C. Binnema                  Helsinki, Finland
e:djcb@djcbsoftware.nl           w:www.djcbsoftware.nl
gpg: 6987 9CED 1745 9375 0F14 DA98 11DD FEA9 DCC4 A036

Re: [PATCH] Project out of sources compilation

[Ergus]

On Mon, Apr 01, 2024 at 10:49:25AM +0300, Dirk-Jan C. Binnema wrote:
>On Sunday Mar 31 2024, Ergus wrote:
>
>> Hi Dmitry:
>>
>> On Sun, Mar 31, 2024 at 05:41:24AM +0300, Dmitry Gutov wrote:
>>>Hi Ergus,
>>>
>>>On 27/03/2024 18:38, Ergus wrote:
>>>
>
>>> I originally suggested a variable that stores a relative file name. But I see
>>> that you prefer the fetching of this file name to be more dynamic.
>>>
>> Yes, that's actually the point. In out of sources compilation there is a
>> possibility that there will be more than one subdirectory to build, in
>> that case the user may be asked at least once which one to choose. This
>> is also valid to detect the `compile-commands.json' which may have
>> different version in each of the build-dir candidates.
>>
>> https://github.com/Ergus/project-multi-mode/blob/c679d73a26b162e4158f6451027ca94e33faca0b/project-multi-mode.el#L109
>
>My 0.02: Having separate build-directories seems to be increasingly
>common (in my bubble I hardly see anything else anymore)
>
Same for me

>Allowing for _multiple_ build-directories is also quit common/useful.
>E.g., for doing debug/release builds, or for building for different
>target hw (embedded dev). For me, a daily need.
>
This is actually part of the main goal

>So having direct support for this in project.el would be quite welcome,
>so I can use that instead of my private hacks.
>
Projectile already have support for many backends and the implementation
is pretty clean. I am doing this because I thing that something so basic
must be part of vanilla.

>>>Why don't we just make it a function variable, at least at first?
>>>
>> Because I wanted somehow to follow the project.el schema of using a
>> generic that the backend could optionally define transparently.
>>
>> And make that definition independent from the user config variable, so,
>> the user defined variable takes preference over the generic search
>> performed by the backend which if not defined automatically goes to
>> project root. Somehow adding a custom variable I think interferes with
>> the idea that the backend provides the functionality... But I may be
>> wrong.
>>
>>> Call it 'project-compile-dir`, which would be settable to a string (constant
>>> value, to be customized through dir-locals), or a function that either accepts
>>> the current project value, or is simply called "inside project root" (with
>>> default-directory bound to that value).
>>>
>> I will try this again, but I had some issues before.
>>
>>> Alternatively, if you see a need to add additional methods
>>> ('project-compile-command'?), it could become a new hook and a new facility
>>> (e.g. project-build-functions) which could be structures similarly to
>>> project.el, i.e. the hook returns a "project builder" value, and the value has
>>> some methods that it can defined/override. Then different build tool backends
>>> could be mixed with different project backends more easily.
>>>
>> I would prefer avoid more abstraction layers in the api. We actually
>> need some `project-compile-command' like function, but structuring it
>> with generics and more layer is IMHO adding too much complexity for a
>> public api... Isn't there a simpler alternative?
>>
>>> So far I only see one additional method, though, and the constructed
>>> compile-command value is relatively simple. Curious to see whether this will
>>> get more complex with additional targets.
>>>
>> I added one method and with that I already support autotools and cmake.
>
>(would be nice to add "meson" too!)
>
No problem once the feature will be more advanced I can add it to the
project-multi backend in my github...

>No immediate opinions on how to implement this, but things I commonly
>want in projects:
>
>- build
>- run
>- test
>- install
>- flash
>- debug
>- clean
>
Lets start with the one common to all: `build'

Some others are difficult to support i.e I have no idea what is
`flash'. And makefile based projects may not have a clean target defined.

The install may be also controversial as it may refer to install in the
system or create an install directory.

And run is complicated because most projects create multiple outputs, so
we need to find some kind of standard method to list them in order to at
least ask the user in the first call.

As Dmitry said, the multiple targets may be also difficult to handle
with this abstraction either to `build only one' (may require completion
in the command manually) or to debug (in order to know which
target/executable to debug). I am not saying it is impossible, just that
I don't have enough knowledge about all the tools and their support for
such commands.

Also remember that this only applies to specific backends because vc
backends know nothing about this, so the commands need to be defined
somehow conditionally in M-x according to the backend... That's very far
from my elisp capabilities...

>It's possible of course to "multiplex" with the compile-command and put
>in a "transient" or something like that; but it'd be nice to handle this
>directly across projects; also for adding menu / toolbar commands.
>
>Guess this is a bit of an open-ended list, so having `project-*-command`
>may not to be the best.
>
Agree here, but I am not a good lisper... so probably Dmitry can bring
some advanced elisp feature I am not be aware of that could solve this issue.

>Kind regards,
>Dirk.
>
>-- 
>Dirk-Jan C. Binnema                  Helsinki, Finland
>e:djcb@djcbsoftware.nl           w:www.djcbsoftware.nl
>gpg: 6987 9CED 1745 9375 0F14 DA98 11DD FEA9 DCC4 A036
>

Re: [PATCH] Project out of sources compilation

[Dirk-Jan C. Binnema]

On Monday Apr 01 2024, Ergus wrote:

> On Mon, Apr 01, 2024 at 10:49:25AM +0300, Dirk-Jan C. Binnema wrote:

>>> I added one method and with that I already support autotools and cmake.
>>
>>(would be nice to add "meson" too!)
>>
> No problem once the feature will be more advanced I can add it to the
> project-multi backend in my github...

Great!

For meson support, I found I needed somewhat of a variation on
`locate-dominating-file' to find the remotest parent directory that has
a meson.build. Guess it's similar to what's needed for cmake. 

>>No immediate opinions on how to implement this, but things I commonly
>>want in projects:
>>
>>- build
>>- run
>>- test
>>- install
>>- flash
>>- debug
>>- clean
>>
> Lets start with the one common to all: `build'
>
> Some others are difficult to support i.e I have no idea what is
> `flash'. And makefile based projects may not have a clean target defined.

Yes, these are only useful for _some_ projects; currently I have some
custom solution to determine the flash command for the current
project (if any). But I think it'd be nice to somehow attach the
functionality to the current project. 

Note - it'd indeed be hard to automatically guess/determine this for
some project; in many cases, it would probably require the user to
configure some project-specific variable to set the commands.

> The install may be also controversial as it may refer to install in the
> system or create an install directory.
>
> And run is complicated because most projects create multiple outputs, so
> we need to find some kind of standard method to list them in order to at
> least ask the user in the first call.

As above, some of this might require some project-specific user
variables.

For comparison e.g. VSCode allows for doing such actions for many
projects; Emacs can do that as well, but currently it requires quite a
bit of custom tweaking; I'm just saying it'd be nice to integrate that
with project.el.

Some .project-locals.el could be useful for this.

> As Dmitry said, the multiple targets may be also difficult to handle
> with this abstraction either to `build only one' (may require
> completion in the command manually) or to debug (in order to know
> which target/executable to debug). I am not saying it is impossible,
> just that I don't have enough knowledge about all the tools and their
> support for such commands.
>
> Also remember that this only applies to specific backends because vc
> backends know nothing about this, so the commands need to be defined
> somehow conditionally in M-x according to the backend... That's very far
> from my elisp capabilities...
>
>>It's possible of course to "multiplex" with the compile-command and put
>>in a "transient" or something like that; but it'd be nice to handle this
>>directly across projects; also for adding menu / toolbar commands.
>>
>>Guess this is a bit of an open-ended list, so having `project-*-command`
>>may not to be the best.
>>
> Agree here, but I am not a good lisper... so probably Dmitry can bring
> some advanced elisp feature I am not be aware of that could solve this
> issue.

Sorry for hi-jacking your useful proposal to propose some more :-)

Kind regards,
Dirk.

-- 
Dirk-Jan C. Binnema                  Helsinki, Finland
e:djcb@djcbsoftware.nl           w:www.djcbsoftware.nl
gpg: 6987 9CED 1745 9375 0F14 DA98 11DD FEA9 DCC4 A036

Re: [PATCH] Project out of sources compilation

[Ergus]

On Mon, Apr 01, 2024 at 06:09:48PM +0300, Dirk-Jan C. Binnema wrote:
>On Monday Apr 01 2024, Ergus wrote:
>
>> On Mon, Apr 01, 2024 at 10:49:25AM +0300, Dirk-Jan C. Binnema wrote:
>
>>>> I added one method and with that I already support autotools and cmake.
>>>
>>>(would be nice to add "meson" too!)
>>>
>> No problem once the feature will be more advanced I can add it to the
>> project-multi backend in my github...
>
>Great!
>
>For meson support, I found I needed somewhat of a variation on
>`locate-dominating-file' to find the remotest parent directory that has
>a meson.build. Guess it's similar to what's needed for cmake.�
>

Yes in project-multi I only need to add a simple entry like this:

(:type cmake                      // will be meson in your case
  :program "cmake"                 // name of program to build
  :root-hint "CMakeLists.txt"      // the filename to search in the root dir (gets the top most if nested)
  :build-hint "CMakeCache.txt"     // The file to search in the build dir (searches in root and all root's subdirs)
  :project-regex "project[[:blank:]]*([[:blank:]]*\\([[:alnum:]]+\\).+)" // Regex to search the project name in :root-hint
  :compile-command ":program --build ."  // A command pattern to build from build-dir; `:program' will be substituted
)

:project-regex can also be a function.

sadly I don't really have time now to implement another version of the
code needed in project.el... Because there is a part in Dmitry's
comments I don't really know how to manage to support the workflow I
have now...


>>>No immediate opinions on how to implement this, but things I commonly
>>>want in projects:
>>>
>>>- build
>>>- run
>>>- test
>>>- install
>>>- flash
>>>- debug
>>>- clean
>>>
>> Lets start with the one common to all: `build'
>>
>> Some others are difficult to support i.e I have no idea what is
>> `flash'. And makefile based projects may not have a clean target defined.
>
>Yes, these are only useful for _some_ projects; currently I have some
>custom solution to determine the flash command for the current
>project (if any). But I think it'd be nice to somehow attach the
>functionality to the current project.
>
>Note - it'd indeed be hard to automatically guess/determine this for
>some project; in many cases, it would probably require the user to
>configure some project-specific variable to set the commands.
>
>> The install may be also controversial as it may refer to install in the
>> system or create an install directory.
>>
>> And run is complicated because most projects create multiple outputs, so
>> we need to find some kind of standard method to list them in order to at
>> least ask the user in the first call.
>
>As above, some of this might require some project-specific user
>variables.
>
>For comparison e.g. VSCode allows for doing such actions for many
>projects; Emacs can do that as well, but currently it requires quite a
>bit of custom tweaking; I'm just saying it'd be nice to integrate that
>with project.el.
>
>Some .project-locals.el could be useful for this.
>
>> As Dmitry said, the multiple targets may be also difficult to handle
>> with this abstraction either to `build only one' (may require
>> completion in the command manually) or to debug (in order to know
>> which target/executable to debug). I am not saying it is impossible,
>> just that I don't have enough knowledge about all the tools and their
>> support for such commands.
>>
>> Also remember that this only applies to specific backends because vc
>> backends know nothing about this, so the commands need to be defined
>> somehow conditionally in M-x according to the backend... That's very far
>> from my elisp capabilities...
>>
>>>It's possible of course to "multiplex" with the compile-command and put
>>>in a "transient" or something like that; but it'd be nice to handle this
>>>directly across projects; also for adding menu / toolbar commands.
>>>
>>>Guess this is a bit of an open-ended list, so having `project-*-command`
>>>may not to be the best.
>>>
>> Agree here, but I am not a good lisper... so probably Dmitry can bring
>> some advanced elisp feature I am not be aware of that could solve this
>> issue.
>
>Sorry for hi-jacking your useful proposal to propose some more :-)
>
Not need to sorry... is actually good that someone else also cares about
this... ;)

>Kind regards,
>Dirk.
>
>-- 
>Dirk-Jan C. Binnema                  Helsinki, Finland
>e:djcb@djcbsoftware.nl           w:www.djcbsoftware.nl
>gpg: 6987 9CED 1745 9375 0F14 DA98 11DD FEA9 DCC4 A036
>

Re: [PATCH] Project out of sources compilation

[Dmitry Gutov]

On 01/04/2024 16:52, Ergus wrote:
>> So having direct support for this in project.el would be quite welcome,
>> so I can use that instead of my private hacks.
>>
> Projectile already have support for many backends and the implementation
> is pretty clean. I am doing this because I thing that something so basic
> must be part of vanilla.

Speaking of Projectile, it has this block of definitions of "project 
types". Which could maybe better called "build tool types" in the 
context of this discussion.

It would be nice if we could port them over to a design which would be 
useable by both Projectile and project.el (and together with other 
project.el backends), rather than having every project backend 
re-implement these settings, or re-enumerate the supported types.

That's why the idea of orthogonal APIs seems appealing to me (the 
current one for file listing and root finding; the new one for build 
tools, and compiling, and running, I guess).

Until we've drawn up some design for the latter, though, maybe a 
customizable function variable will suffice.

Re: [PATCH] Project out of sources compilation

[Ergus]

Hi Dmitry:

So far I Think I understood part of your las email about functionality.

On Wed, Apr 03, 2024 at 02:23:03AM +0300, Dmitry Gutov wrote:
>On 01/04/2024 16:52, Ergus wrote:
>>>So having direct support for this in project.el would be quite welcome,
>>>so I can use that instead of my private hacks.
>>>
>>Projectile already have support for many backends and the implementation
>>is pretty clean. I am doing this because I thing that something so basic
>>must be part of vanilla.
>
>Speaking of Projectile, it has this block of definitions of "project 
>types". Which could maybe better called "build tool types" in the 
>context of this discussion.
>
>It would be nice if we could port them over to a design which would be 
>useable by both Projectile and project.el (and together with other 
>project.el backends), rather than having every project backend 
>re-implement these settings, or re-enumerate the supported types.
>
Their design is actually fine, but I would prefer to keep things simpler
in the api side because project is intended to be more general.

Their use case adds too many functions in order to register, call, test
and generate the different backends I thing it is actually too much for
vanilla. If we want that complexity maybe is better to retake the last
year discussion you mentioned before:

https://lists.gnu.org/archive/html/emacs-devel/2023-05/msg00509.html

But adding that will require also parallel work and code in order to
integrate with project.el.

>That's why the idea of orthogonal APIs seems appealing to me (the 
>current one for file listing and root finding; the new one for build 
>tools, and compiling, and running, I guess).
>
I would probably insist in a simpler basic feature still in the current
project.el. I can't see the features as a separated thing.

What I would probably propose here is (maybe) to move the vc-backend to
it's own file as a separated package in order to keep the general api
separated from the "plugin". So if a developer wants to create a backend
it is clear whats needed and what the standard implementation example...

>Until we've drawn up some design for the latter, though, maybe a 
>customizable function variable will suffice.
>
What I have in mind as now is to add a variable and a method that can be
both optionally defined.

The method defined by backends and the variable by the user (maybe in
dir-locals).

This way the backends won't be forced to define any of these if
compilation is not supported (i.e vs-backend) or if the project is for a
non-compilable languaje (i.e an hypothetic python project)

project-compile-info accepts a second variable that in this case expects
'dir or 'compile-cmd in any case when not implemented the function gives
null and the caller (compile in this case) falls back to the default
values.

This can accept any other command like 'generate-cmd 'test-cmd and so
on.  In the future we can define any other random command relying on
these

So far the only we need for compilation is more or less this:

```
diff --git a/lisp/progmodes/project.el b/lisp/progmodes/project.el
index a10e24f3e28..9d77d5452f3 100644
--- a/lisp/progmodes/project.el
+++ b/lisp/progmodes/project.el
@@ -290,8 +290,29 @@ project-external-roots
  headers search path, load path, class path, and so on."
    nil)
  
+(cl-defgeneric project-compile-info (_project _info)
+  "Return build information of the current PROJECT.
+
+This function is intended to be defined by the backend when possible.
+Otherwise this returns nil and the `project-compile' command will use
+the default values.
+The current valid values for INFO are `dir' and `compile-cmd'"
+  nil)
+
+(defun project-get-build-dir (project)
+  "Return absolute build directory for the current PROJECT.
+1. Tries the generic function `project-compile-info' with info='dir.
+2. else it return the project-root.
+If the defined path is relative, this expands it relatively to the
+project's root."
+  (let ((dir (or (project-compile-info project 'dir)   ;; backend function
+                 (project-root project))))        ;; I assume project-root is always absolute
+    (if (file-name-absolute-p dir)
+        dir
+      (expand-file-name dir (project-root project)))))


@@ -1392,10 +1413,15 @@ project-compile
    "Run `compile' in the project root."
    (declare (interactive-only compile))
    (interactive)
-  (let ((default-directory (project-root (project-current t)))
-        (compilation-buffer-name-function
-         (or project-compilation-buffer-name-function
-             compilation-buffer-name-function)))
+  ;; I am wondering whenever we need to expand connection local
+  ;; variables at this point... maybe before or inside the let.
+  (let* ((project (project-current t))
+         (default-directory (project-get-build-dir project))
+         (compile-command (or (project-compile-info project 'compile-cmd)
+                              compile-command))
+         (compilation-buffer-name-function
+          (or project-compilation-buffer-name-function
+              compilation-buffer-name-function)))
      (call-interactively #'compile)))

```

Finally the only missing detail is user side configuration... I was
thinking in a plist defconst (similar to the approach in
eglot-workspace-configuration) so the user can set it completelly or
partially and we only need to do:

(or (plist-get project-info-plist 'compile-cmd)
     (project-compile-info project 'compile-cmd)
     compile-command)

In this way the user only has one variable to set in the config with all
the parameters optional as well which stays easy, simple in
implementation and with a predictable behavior.

WDYT??
Best,
Ergus

Re: [PATCH] Project out of sources compilation

[Ergus]

[-- Attachment #1: Type: text/plain, Size: 791 bytes --]

Hi Dmitry:

Here I attach a new patch with what I think is the best approach to
handle the extra info for projects. This implementation is pretty simple
for both sides (user, project.el and backends) and uses your main idea
of the generics + a custom.

However I decided not to use only the custom as you suggested, because
considering the project initialization it will require a complex and
error prone approach to initalize the backend variables correctly (for
example, when some buffers are already open).

With this implementation the backends only need to define the optional
generic specializations and the policy is pretty simple to explain (see
the doc strings).

I also modifies the backend project on github to match this new api, so,
you can check it as well.

WDYT?
Best,
Ergus

[-- Attachment #2: project-extra-info.patch --]
[-- Type: text/plain, Size: 5481 bytes --]

diff --git a/lisp/progmodes/project.el b/lisp/progmodes/project.el
index da211566a3b..1986c14e861 100644
--- a/lisp/progmodes/project.el
+++ b/lisp/progmodes/project.el
@@ -51,6 +51,16 @@
 ;; files inside the root must not be considered a part of it).  It
 ;; should be consistent with `project-files'.
 ;;
+;; `project-extra-info' is a user custom variable and a method to
+;; specify extra project information like build command or equivalent.
+;; The variable is a plist were the values can be strings or functions
+;; that receive the project-current as argument.  The
+;; `project-extra-info' expect project current as first parameter and an
+;; extra parameter equivalent to the plist key.  The user defined plist
+;; takes precedence over the backend defined methods, but all the
+;; specializations are optional and the functions calling them may
+;; provide conditions in case both are undefined.
+;;
 ;; This list can change in future versions.
 ;;
 ;; Transient project:
@@ -229,8 +239,8 @@ project-current
 of the project instance object."
   (unless directory (setq directory (or project-current-directory-override
                                         default-directory)))
-  (let* ((non-essential (not maybe-prompt))
-         (pr (project--find-in-directory directory)))
+  (let ((pr (project--find-in-directory directory))
+        (non-essential (not maybe-prompt)))
     (cond
      (pr)
      ((unless project-current-directory-override
@@ -290,8 +300,54 @@ project-external-roots
 headers search path, load path, class path, and so on."
   nil)
 
+(defcustom project-extra-info nil
+  "Project extra info defined in user space.
+
+This is intended to be set by the user.  This is expected to be a plist
+with key entries for compile.  At the moment the implemented keys are
+`:compile-command' and `:compile-dir'.  The entries may be either string
+constants, paths or functions.  This custom has a symmetric generic
+method with the same name that are intended to be implemented by the
+project backends.  When this variable is defined it takes precedence
+over the backend methods."
+  :safe t
+  :version "30.1"
+  :type '(plist :key-type (choice (const :compile-dir)
+                                  (const :compile-command))
+                :value-type (choice string
+                                    directory
+                                    function)))
+
+(cl-defgeneric project-extra-info (_project _info)
+  "Return extra INFO for the current PROJECT.
+
+This function is intended to be defined by the backend when needed.
+Otherwise this returns nil and the `project-compile' command will use
+some default values.  The current valid values for INFO are the same key
+types in project-compile-info: `:compile-dir' and `:compile-command'
+This method is independent from the custom variable with same name
+because project.el initializes itself lazily and variable propagation
+within directories and buffers already open will require too much work
+in the user side potentially more error prone."
+  nil)
+
+(defun project--get-extra-info (project info)
+  "Steps to get PROJECT's INFO internally.
+1. Parse the user defined variable `project-compile-info'.  If the key
+exists:
+   a. Check if it is a function and call it passing project as the first
+parameter.
+   b. If the key is a string return it as is.
+   c. Otherwise return nil.
+2. Else call the backend defined method `project-compile-info'."
+  (if-let ((value (plist-get project-extra-info info)))
+      (cond ((functionp value) (funcall value project))
+            ((stringp info) info)
+            (t nil))
+    (project-extra-info project info)))
+
 (cl-defgeneric project-name (project)
-  "A human-readable name for the project.
+  "A human-readable name for the PROJECT.
 Nominally unique, but not enforced."
   (file-name-nondirectory (directory-file-name (project-root project))))
 
@@ -1389,13 +1445,26 @@ project-compilation-buffer-name-function
 
 ;;;###autoload
 (defun project-compile ()
-  "Run `compile' in the project root."
+  "Run `compile' in the project build directory.
+When the variable `project-extra-info' contains the entries
+`:compile-dir' or `:compile-command' or the project backend specializes
+the method `project-extra-info' for those values; then this command uses
+that instead of the default: `project-root' and `compile-command'."
   (declare (interactive-only compile))
   (interactive)
-  (let ((default-directory (project-root (project-current t)))
-        (compilation-buffer-name-function
-         (or project-compilation-buffer-name-function
-             compilation-buffer-name-function)))
+  ;; I am wondering whenever we need to expand connection local
+  ;; variables at this point... maybe before or inside the let.
+  (let* ((project (project-current t))
+         (dir (or (project--get-extra-info project :compile-dir)
+                  (project-root project)))
+         (default-directory (if (file-name-absolute-p dir)
+                                dir
+                              (expand-file-name dir (project-root project))))
+         (compile-command (or (project--get-extra-info project :compile-command)
+                              compile-command))
+         (compilation-buffer-name-function
+          (or project-compilation-buffer-name-function
+              compilation-buffer-name-function)))
     (call-interactively #'compile)))
 
 (defun project-recompile (&optional edit-command)

Re: [PATCH] Project out of sources compilation

[Dmitry Gutov]

[-- Attachment #1: Type: text/plain, Size: 1020 bytes --]

Hi Ergus!

On 06/04/2024 05:05, Ergus wrote:

> Here I attach a new patch with what I think is the best approach to
> handle the extra info for projects. This implementation is pretty simple
> for both sides (user, project.el and backends) and uses your main idea
> of the generics + a custom.
> 
> However I decided not to use only the custom as you suggested, because
> considering the project initialization it will require a complex and
> error prone approach to initalize the backend variables correctly (for
> example, when some buffers are already open).
> 
> With this implementation the backends only need to define the optional
> generic specializations and the policy is pretty simple to explain (see
> the doc strings).
> 
> I also modifies the backend project on github to match this new api, so,
> you can check it as well.

How about the idea with the orthogonal hook, then, as in the attached 
patch? Something I described before.

I think your github code would be easy enough to adapt to use it.

WDYT?

[-- Attachment #2: project-build-functions.diff --]
[-- Type: text/x-patch, Size: 1996 bytes --]

diff --git a/lisp/progmodes/project.el b/lisp/progmodes/project.el
index 000a05804a8..bd1264b5525 100644
--- a/lisp/progmodes/project.el
+++ b/lisp/progmodes/project.el
@@ -1387,15 +1387,33 @@ project-compilation-buffer-name-function
                         project-prefixed-buffer-name)
                  (function :tag "Custom function")))
 
+(defvar project-build-functions nil
+  "Special hook to find the project build configuration.
+Each functions on this hook is called in turn with one
+argument (project), and should return either nil to mean that it is not
+applicable, or a plist with build configuration.
+
+The configuration can contain keys such as `:compile-dir',
+`:compile-command', or others.")
+
 ;;;###autoload
 (defun project-compile ()
-  "Run `compile' in the project root."
+  "Run `compile' in the project build directory.
+Uses the `project-build-functions' hook to determine the compile command
+and the directory to run it in."
   (declare (interactive-only compile))
   (interactive)
-  (let ((default-directory (project-root (project-current t)))
-        (compilation-buffer-name-function
-         (or project-compilation-buffer-name-function
-             compilation-buffer-name-function)))
+  (let* ((project (project-current t))
+         (build-config (run-hook-with-args 'project-build-functions project))
+         (default-directory (if (plist-get build-config :compile-dir)
+                                (expand-file-name (plist-get build-config :compile-dir)
+                                                  (project-root project))
+                              default-directory))
+         (compile-command (or (plist-get build-config :compile-command)
+                              compile-command))
+         (compilation-buffer-name-function
+          (or project-compilation-buffer-name-function
+              compilation-buffer-name-function)))
     (call-interactively #'compile)))
 
 (defun project-recompile (&optional edit-command)

Re: [PATCH] Project out of sources compilation

[Ergus]

Hi Dmitry:

Sorry for th delay with this

On Sun, Apr 14, 2024 at 04:44:36AM +0300, Dmitry Gutov wrote:
>Hi Ergus!
>
>On 06/04/2024 05:05, Ergus wrote:
>
>>Here I attach a new patch with what I think is the best approach to
>>handle the extra info for projects. This implementation is pretty simple
>>for both sides (user, project.el and backends) and uses your main idea
>>of the generics + a custom.
>>
>>However I decided not to use only the custom as you suggested, because
>>considering the project initialization it will require a complex and
>>error prone approach to initalize the backend variables correctly (for
>>example, when some buffers are already open).
>>
>>With this implementation the backends only need to define the optional
>>generic specializations and the policy is pretty simple to explain (see
>>the doc strings).
>>
>>I also modifies the backend project on github to match this new api, so,
>>you can check it as well.
>
>How about the idea with the orthogonal hook, then, as in the attached 
>patch? Something I described before.
>
>I think your github code would be easy enough to adapt to use it.
>
>WDYT?

The main idea is somehow the same, but IMHO I think that this approach
complicates the user config and backend implementation being also less
efficient and scalable (we will need a new hook for every new similar
feature).

There is no need to separate the backend from the build, and after some
tests, I find the cl-defgeneric behavior more predictable and simpler to
debug than using a function hook.

Your initial idea implemented like 

(cl-defgeneric project-extra-info (_project _info)
   ...)

In my opinion is simpler and more scalable (while could be recycled to
provide also other existing functionalities like
project-compilation-buffer-name-function. All the generic
implementations become then optional which is very desired for the
general purpose.

Re: [PATCH] Project out of sources compilation

[Ergus]

Hi Juri:

Give a look to this:

https://github.com/mohkale/projection

It seems similar to what we may consider to support in vanilla. OR at
least provide the interactive interface wrapper functions in order to
make project.el a bit more standard.

On Sun, Apr 14, 2024 at 04:44:36AM GMT, Dmitry Gutov wrote:
>Hi Ergus!
>
>On 06/04/2024 05:05, Ergus wrote:
>
>>Here I attach a new patch with what I think is the best approach to
>>handle the extra info for projects. This implementation is pretty simple
>>for both sides (user, project.el and backends) and uses your main idea
>>of the generics + a custom.
>>
>>However I decided not to use only the custom as you suggested, because
>>considering the project initialization it will require a complex and
>>error prone approach to initalize the backend variables correctly (for
>>example, when some buffers are already open).
>>
>>With this implementation the backends only need to define the optional
>>generic specializations and the policy is pretty simple to explain (see
>>the doc strings).
>>
>>I also modifies the backend project on github to match this new api, so,
>>you can check it as well.
>
>How about the idea with the orthogonal hook, then, as in the attached 
>patch? Something I described before.
>
>I think your github code would be easy enough to adapt to use it.
>
>WDYT?

Re: [PATCH] Project out of sources compilation

[Ergus]

Adding Mohsin Kaleem to the discussion.


On Mon, Apr 22, 2024 at 07:05:20PM GMT, Ergus wrote:
>Hi Juri:
>
>Give a look to this:
>
>https://github.com/mohkale/projection
>
>It seems similar to what we may consider to support in vanilla. OR at
>least provide the interactive interface wrapper functions in order to
>make project.el a bit more standard.
>
>On Sun, Apr 14, 2024 at 04:44:36AM GMT, Dmitry Gutov wrote:
>>Hi Ergus!
>>
>>On 06/04/2024 05:05, Ergus wrote:
>>
>>>Here I attach a new patch with what I think is the best approach to
>>>handle the extra info for projects. This implementation is pretty simple
>>>for both sides (user, project.el and backends) and uses your main idea
>>>of the generics + a custom.
>>>
>>>However I decided not to use only the custom as you suggested, because
>>>considering the project initialization it will require a complex and
>>>error prone approach to initalize the backend variables correctly (for
>>>example, when some buffers are already open).
>>>
>>>With this implementation the backends only need to define the optional
>>>generic specializations and the policy is pretty simple to explain (see
>>>the doc strings).
>>>
>>>I also modifies the backend project on github to match this new api, so,
>>>you can check it as well.
>>
>>How about the idea with the orthogonal hook, then, as in the 
>>attached patch? Something I described before.
>>
>>I think your github code would be easy enough to adapt to use it.
>>
>>WDYT?
>
>
>

Re: [PATCH] Project out of sources compilation

[Mohsin Kaleem]

Ergus <spacibba@aol.com> writes:

Howdie howdie,

I'm the maintainer of projection. Looks like it's a similar external
package to what we're trying to formalize in Emacs core project.el .

I'm just going to highlight 2 things which I think would help you Ergus.
Ninja multi-config generators [1] which basically lets you configure a
project for Debug, Release, ASAN, UBSAN, etc. simultaneously. And CMake
presets [2] which is a standard for loading common configuration in CMake
like build directories but also compiler flags, environment variables,
etc.

[1]: https://cmake.org/cmake/help/latest/generator/Ninja%20Multi-Config.html
[2]: https://cmake.org/cmake/help/latest/manual/cmake-presets.7.html

I'll add my 2-cents on some of the discussed points.

Firstly, I disagree with the goal of making the build directory option
customizable in project.el itself. Most build tools I've encountered
(including all supported by projection) take an extra flag to point to
the build directory and I find this a nicer UX because it's easily
inspectable and editable with `C-u M-x recompile`. I noticed projectile
supported this feature of "run somewhere else" as well but it was only
for one project type "meson" and even that has this option to move
directories so I don't really see the value of complicating the
interface to support it. Another pain point with a generalised
"build-directory" option is that it's not generic for all possible
commands. Configuration often runs before the build directory exists
(because it creates it) so then you need an exemption for one command
type but not another. I think the project-root as a standard location
for commands to run is a good common default and changing it unless
absolutely necessary is un-needed :-).

Secondly I agree with Dmitry about defining an alternate orthoganal API
to project.el for this. It probably belongs more in a project-types.el
package. The motivation being that project.el is for more bare-bones
mechanics, things like "am I in a project", "where is the root of the
project", "show me all the files in this project". Project types is more
"how do I build this project", "how do I test it", "what artifacts does it
produce". The former is a set of functionalities which is relatively
common and static across all projects, the latter is very specific to
what kind of a project we're dealing with and how you interface with it.
For example python projects probably don't have a configure step or
build directory, node projects may treat "npm install" as a configure
step but omit building, CMake projects support all steps through to
packaging and installing. Things don't generalise well here.

I think Ergus you've actually touched on a fun corner case. You've got a
project with sub-projects and define a regex in a marker file to detect
this. I'll admit I haven't touched on this before, I know you can have a
CMake project with different sub-directories being considered their own
projects and buildable independently, but I've never seen this practice
in my day-to-day work. I'd be inclined to treat this as a project.el
feature, something akin to git sub-modules. My only concern with it as a
feature is its not trivial to check like git submodules. Basically every
directory in a CMake project can have a CMakeLists.txt file and
recursively searching upwards for one that might coincidentally also be
a project root marker isn't ideal :-(.

While we're discussing interfaces for this project-type abstraction I'll
mention what I did for projection. In projection a project-type is an
eioio class. It has some slots like :predicate to check if the current
project matches that type and :build, :test, etc. for various project
specific commands that can be run. The list of all actively checked-for
project types is just a flat hook. I opted for this for 2 reasons:

1. It's easy to modify. Every project-type can be edited directly you
can deregistered one from the monitored set of types by just removing it
from the collection.
2. It enforces a general ordering concept which is important for
determining a primary project type. I think moving the properties into a
generic function though would just decouple them and make it harder to
reason about and modify them.

In regards to interactivity, I outsourced a lot to another package I
maintain called compile-multi [3]. At its core its just a menu for
predicates and compile commands. When predicate X, you have the option
to choose command Y. I've plugged it into projection so that through it
you can see project type specific options. For example in a CMake
project this menu shows you all compilation targets and all CTest
targets. Running a selected target will run the compile command that
builds that target. If you want to permanently override one that runs
with the standard projection-commands-build-project or
projection-commands-test-project interactive commands (this is the same
as setting what you're wanting to build going forward) I recommend using
of embark [4]. What I normally do is run `M-x projection-multi-compile`
narrow to the build or test target I care about and then run `M-x
embark-act p c` (or p t for a test) and next time I run
projection-commands-build-project it'll use that selected command. If I
want to reset this to the project default I can use `M-x
projection-cache-clear` which shows all cached project options and lets
me select one or more to prune. Next time I run any command it'll be
the default for the current project.

[3]: https://github.com/mohkale/compile-multi
[4]: https://github.com/oantolin/embark

I think generalising this latter feature to a common interface will be
tricky. To begin with just knowing a project has build target X doesn't
really tell you how to build X, just that you can. You need to feed that
selection back into the project-type or export it directly as is. I
recently added a feature to list build artifacts (for debugger
integration) and that shows just how much variety there is here.

Lastly, just a general point (sorry for writing so much), I think any
feature for this shouldn't assume only a single project type is valid.
There should be a concept of a primary type just to make commands like
configure or build make sense relative to each other, but nothing stops
a project from matching two types at once and at least IME that's the
more natural state of things. I commonly work on C++ projects but we
also have a package.json in those projects file so we can depend on
prettier and other non-C++ specific linters. The way projection gets
around this is by always matching all defined project types. Users can
add to the list of matched types for the current project and can cycle
the primary project type. Certain interfaces that aren't single project
specific like the aforementioned projection-multi-compile will source
compilation targets from all applicable project types. This provides an
extremely rich interface for interacting with projects (at least IMO).

Let me know if anyone has any thoughts or questions about projection.
I'm not averse to re-aligning across something in Emacs mainline. The
only minor concern is how specific it is to the kinds of projects I work
on and that may not generalise well to others.

-- 
Mohsin Kaleem

Re: [PATCH] Project out of sources compilation

[Ergus]

Hi:

On Mon, Apr 22, 2024 at 10:20:32PM GMT, Mohsin Kaleem wrote:
>Ergus <spacibba@aol.com> writes:
>
>Howdie howdie,
>
>I'm the maintainer of projection. Looks like it's a similar external
>package to what we're trying to formalize in Emacs core project.el .
>
>I'm just going to highlight 2 things which I think would help you Ergus.
>Ninja multi-config generators [1] which basically lets you configure a
>project for Debug, Release, ASAN, UBSAN, etc. simultaneously. And CMake
>presets [2] which is a standard for loading common configuration in CMake
>like build directories but also compiler flags, environment variables,
>etc.
>
>[1]: https://cmake.org/cmake/help/latest/generator/Ninja%20Multi-Config.html
>[2]: https://cmake.org/cmake/help/latest/manual/cmake-presets.7.html
>
I'll give it a look. I am aware of cmake presets, but we don't even
support cmake or out of sources compilation yet, so I tink that
supporting the most basic cmake setup will be already a step forward.

>I'll add my 2-cents on some of the discussed points.
>
>Firstly, I disagree with the goal of making the build directory option
>customizable in project.el itself. Most build tools I've encountered
>(including all supported by projection) take an extra flag to point to
>the build directory and I find this a nicer UX because it's easily
>inspectable and editable with `C-u M-x recompile`. I noticed projectile
>supported this feature of "run somewhere else" as well but it was only
>for one project type "meson" and even that has this option to move
>directories so I don't really see the value of complicating the
>interface to support it. Another pain point with a generalised
>"build-directory" option is that it's not generic for all possible
>commands. Configuration often runs before the build directory exists
>(because it creates it) so then you need an exemption for one command
>type but not another. I think the project-root as a standard location
>for commands to run is a good common default and changing it unless
>absolutely necessary is un-needed :-).
>
Yes, that's actually the current approach in my POC code. after some
fight with rust projects. However, the difference on moving or not is so
simple that it doesn't even represent a difficulty.

>Secondly I agree with Dmitry about defining an alternate orthoganal API
>to project.el for this. It probably belongs more in a project-types.el
>package. The motivation being that project.el is for more bare-bones
>mechanics, things like "am I in a project", "where is the root of the
>project", "show me all the files in this project". Project types is more
>"how do I build this project", "how do I test it", "what artifacts does it
>produce". The former is a set of functionalities which is relatively
>common and static across all projects, the latter is very specific to
>what kind of a project we're dealing with and how you interface with it.
>For example python projects probably don't have a configure step or
>build directory, node projects may treat "npm install" as a configure
>step but omit building, CMake projects support all steps through to
>packaging and installing. Things don't generalise well here.
>
This implies removing many of the features project.el already has
(project-compile, project-eshell, project-compile-buffer-name-function)
+ somehow create another level of abstraction over project.el...

The initial basic commands (config, build, test/run) seems to be common,
even in python projects it usually needs to get dependencies, create
virtualenvs and so on. However if a project does not support a build
command a simple message can solve it easily.

Initially we are not even supporting the generate/configure, and the
functionality is limited to: recognize compile and test already generated
projects; but using some very primitive heuristics. More complex setups
I thing must be left to more specialized external projects.

These seems limiting, but it is much more than what we already have now.

>I think Ergus you've actually touched on a fun corner case. You've got a
>project with sub-projects and define a regex in a marker file to detect
>this. I'll admit I haven't touched on this before, I know you can have a
>CMake project with different sub-directories being considered their own
>projects and buildable independently, but I've never seen this practice
>in my day-to-day work. I'd be inclined to treat this as a project.el
>feature, something akin to git sub-modules. My only concern with it as a
>feature is its not trivial to check like git submodules. Basically every
>directory in a CMake project can have a CMakeLists.txt file and
>recursively searching upwards for one that might coincidentally also be
>a project root marker isn't ideal :-(.
>
FWIK every directory may have a CMakeLists.txt, but they need to have a
project() entry to be considered a project's CMake. The search is
actually unavoidable and the regex check actually solves the race
conditions.

I know that the search is not efficient, but with a couple of
optimizations if works very well even with tramp.

Nested projects is something very common these days and supporting them
in project.el is part of another parallel discussion going on already
(very slowly btw).

For now it is save enough to find the top-most one as it is what the
default backend does. It will probably fit a high percent of all
projects around.

>While we're discussing interfaces for this project-type abstraction I'll
>mention what I did for projection. In projection a project-type is an
>eioio class. It has some slots like :predicate to check if the current
>project matches that type and :build, :test, etc. for various project
>specific commands that can be run. The list of all actively checked-for
>project types is just a flat hook. I opted for this for 2 reasons:
>
>1. It's easy to modify. Every project-type can be edited directly you
>can deregistered one from the monitored set of types by just removing it
>from the collection.
>2. It enforces a general ordering concept which is important for
>determining a primary project type. I think moving the properties into a
>generic function though would just decouple them and make it harder to
>reason about and modify them.
>

More or less the same idea; I use with plist. I think they a bit
simpler, but the same underground. I tend to avoid eioio because I have
heard some performance complains (don't know if they are justified) but
also because plists are somehow more flexible in some aspects.

>In regards to interactivity, I outsourced a lot to another package I
>maintain called compile-multi [3]. At its core its just a menu for
>predicates and compile commands. When predicate X, you have the option
>to choose command Y. I've plugged it into projection so that through it
>you can see project type specific options. For example in a CMake
>project this menu shows you all compilation targets and all CTest
>targets. Running a selected target will run the compile command that
>builds that target. If you want to permanently override one that runs
>with the standard projection-commands-build-project or
>projection-commands-test-project interactive commands (this is the same
>as setting what you're wanting to build going forward) I recommend using
>of embark [4]. What I normally do is run `M-x projection-multi-compile`
>narrow to the build or test target I care about and then run `M-x
>embark-act p c` (or p t for a test) and next time I run
>projection-commands-build-project it'll use that selected command. If I
>want to reset this to the project default I can use `M-x
>projection-cache-clear` which shows all cached project options and lets
>me select one or more to prune. Next time I run any command it'll be
>the default for the current project.
>
>[3]: https://github.com/mohkale/compile-multi
>[4]: https://github.com/oantolin/embark
>
Actually I am not trying to do anything so fancy. We cannot go from 0 to
100 in a single pass. If we start 1. recognizing projects backends and
giving a better compile-command for them in the right place and to the
right build-dir is a good step forward compared to what we have. Then we
talk about test and finally we can consider the generate one.

>I think generalising this latter feature to a common interface will be
>tricky. To begin with just knowing a project has build target X doesn't
>really tell you how to build X, just that you can. You need to feed that
>selection back into the project-type or export it directly as is. I
>recently added a feature to list build artifacts (for debugger
>integration) and that shows just how much variety there is here.
>
>Lastly, just a general point (sorry for writing so much), I think any
>feature for this shouldn't assume only a single project type is valid.
>There should be a concept of a primary type just to make commands like
>configure or build make sense relative to each other, but nothing stops
>a project from matching two types at once and at least IME that's the
>more natural state of things. I commonly work on C++ projects but we
>also have a package.json in those projects file so we can depend on
>prettier and other non-C++ specific linters. The way projection gets
>around this is by always matching all defined project types. Users can
>add to the list of matched types for the current project and can cycle
>the primary project type. Certain interfaces that aren't single project
>specific like the aforementioned projection-multi-compile will source
>compilation targets from all applicable project types. This provides an
>extremely rich interface for interacting with projects (at least IMO).
>
At the moment project.el only handles one backend at the time. I made a
simple workaround to extend it a bit, but that is a bit tricky and at
this point simplicity matters. Otherwise we end up with an inefficient
and complex interface easy to break and hard to make it work.

>Let me know if anyone has any thoughts or questions about projection.
>I'm not averse to re-aligning across something in Emacs mainline. The
>only minor concern is how specific it is to the kinds of projects I work
>on and that may not generalise well to others.
>
>-- 
>Mohsin Kaleem
>
Ergus

Re: [PATCH] Project out of sources compilation

[Mohsin Kaleem]

Ergus <spacibba@aol.com> writes:

> I'll give it a look. I am aware of cmake presets, but we don't even
> support cmake or out of sources compilation yet, so I tink that
> supporting the most basic cmake setup will be already a step forward.

We being the projects you work on? My suggestion was more for automating
how you set build configurations. Supporting that in Emacs is separate
but still useful. It's just which directory you configure to and build
in is yet another part of this instead of some custom thing only Emacs
understands.

> Yes, that's actually the current approach in my POC code. after some
> fight with rust projects. However, the difference on moving or not is so
> simple that it doesn't even represent a difficulty.

Just moving is easy, but a interface for that that isn't confusing would
probably be a lot harder. For example you can switch to a compilation
buffer and run cd and recompile and that should work. But it's much
easier to just see the command containing the non-common directory it
needs to run in.

> This implies removing many of the features project.el already has
> (project-compile, project-eshell, project-compile-buffer-name-function)
> + somehow create another level of abstraction over project.el...

I think we probably should. There's a lot of builtin Emacs functionality
which is just duplicating standard things in subtly different ways.
project-other-{window,tab}* for example. Embark at least supercedes this
IMO because it generalises the selection backend from the end user
action. In this case project-compile, project-eshell, etc. If this was
being done today I'd probably just define a 'project-run-from-root'
helper and let the end users alias it to a command if they think its
useful. Like so:

  (defalias 'project-compile! (project-run-from-root #'compile))

But I digress.

> The initial basic commands (config, build, test/run) seems to be common,
> even in python projects it usually needs to get dependencies, create
> virtualenvs and so on. However if a project does not support a build
> command a simple message can solve it easily.

I agree, but my point was this doesn't really align with anything else
project.el does atm. Listing files for a project is orthoganal to which
"configure command" makes sense for whatever kind of project this looks
like. And necessarily multiple configure commands may make sense at once.

> FWIK every directory may have a CMakeLists.txt, but they need to have a
> project() entry to be considered a project's CMake. The search is
> actually unavoidable and the regex check actually solves the race
> conditions.
>
> I know that the search is not efficient, but with a couple of
> optimizations if works very well even with tramp.

This aligns with what I know but also that a encompassing CMake with a
project can depend on a sub-directories CMake with a project. So if you
just configure and build the parent project it'll configure and build
all sub-projects. I'm not sure when you'd not want to do this, if the
projects have overlapping targets perhaps... but at that point you
should probably just move them to independent projects because their
already that.

> Nested projects is something very common these days and supporting them
> in project.el is part of another parallel discussion going on already
> (very slowly btw).

I have yet to encounter any so I'll assume you're right :-).

> More or less the same idea; I use with plist. I think they a bit
> simpler, but the same underground. I tend to avoid eioio because I have
> heard some performance complains (don't know if they are justified) but
> also because plists are somehow more flexible in some aspects.

I haven't noticed any performance issues at least in my limited usage of them.

> Actually I am not trying to do anything so fancy. We cannot go from 0 to
> 100 in a single pass. If we start 1. recognizing projects backends and
> giving a better compile-command for them in the right place and to the
> right build-dir is a good step forward compared to what we have. Then we
> talk about test and finally we can consider the generate one.

You should probably keep in mind sometimes there's a feed forward
mechanism in these things. CMake for example requires you to setup a
file prior to configuring if you want to introspect about the build
configuration and available targets. It might help to keep this in mind,
in my case I had to add a hook mechanism for all of these commands after
the fact to generate and keep this file in sync.

[1]: https://cmake.org/cmake/help/latest/manual/cmake-file-api.7.html

> At the moment project.el only handles one backend at the time. I made a
> simple workaround to extend it a bit, but that is a bit tricky and at
> this point simplicity matters. Otherwise we end up with an inefficient
> and complex interface easy to break and hard to make it work.

This is probably another reason this should be orthogonal to project.el.
The project backend is something well defined like a git repo. The
project type backend is whatever arbitrary marker communicates what the
project type is.

-- 
Mohsin Kaleem

Re: [PATCH] Project out of sources compilation

[Dmitry Gutov]

Hi Mohsin,

On 23/04/2024 00:20, Mohsin Kaleem wrote:
> Lastly, just a general point (sorry for writing so much), I think any
> feature for this shouldn't assume only a single project type is valid.
> There should be a concept of a primary type just to make commands like
> configure or build make sense relative to each other, but nothing stops
> a project from matching two types at once and at least IME that's the
> more natural state of things. I commonly work on C++ projects but we
> also have a package.json in those projects file so we can depend on
> prettier and other non-C++ specific linters. The way projection gets
> around this is by always matching all defined project types. Users can
> add to the list of matched types for the current project and can cycle
> the primary project type. Certain interfaces that aren't single project
> specific like the aforementioned projection-multi-compile will source
> compilation targets from all applicable project types. This provides an
> extremely rich interface for interacting with projects (at least IMO).

That's a great point.

And your project looks very nice. I wonder how we could advertise it 
better, so users could have a quicker access to build tool integration.

As a next step for this discussion, I'm curious what's missing in 
"projection" (nice name, though not very discoverable, alas) from 
Ergus's POV. I.e. how far is it from supporting his workflows.

Re: [PATCH] Project out of sources compilation

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

What does "Project out of sources compilation" mean?
I am not sure how to parse that, let alone determine what it signfies.
-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [PATCH] Project out of sources compilation

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 529 bytes --]


Richard Stallman <rms@gnu.org> writes:

> What does "Project out of sources compilation" mean?
> I am not sure how to parse that, let alone determine what it signfies.

That usually means that the compilation happens in a directory which is
not the source directory (similar to how make distcheck does it).

In this case that the build command is issued in a dedicated build
directory which is not the sources root.

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein,
ohne es zu merken.
draketo.de

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Re: [PATCH] Project out of sources compilation

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > What does "Project out of sources compilation" mean?
  > > I am not sure how to parse that, let alone determine what it signfies.

  > That usually means that the compilation happens in a directory which is
  > not the source directory (similar to how make distcheck does it).

Thanks.  I've been building Emacs that way for half my life,
but I could not tell that the phrase should be parsed that way.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: [PATCH] Project out of sources compilation

[Konstantin Kharlamov]

On Tue, 2024-04-02 at 17:39 -0400, Richard Stallman wrote:
> [[[ To any NSA and FBI agents reading my email: please consider   
> ]]]
> [[[ whether defending the US Constitution against all enemies,    
> ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example.
> ]]]
> 
> What does "Project out of sources compilation" mean?
> I am not sure how to parse that, let alone determine what it
> signfies.

In Emacs terms, if you enter Emacs repo and execute

	mkdir build && cd build && ../configure && make

it will be an "out of sources compilation", because all artifacts¹ will
be inside this `build` dir rather than being scattered all over the
repo.

1: Emacs does not fully support it though, because while object files
and configs do end up inside `build`, however `*.elc` files are still
scattered all over the repo. But I hope you get the idea.

Re: [PATCH] Project out of sources compilation

[Eli Zaretskii]

> From: Konstantin Kharlamov <Hi-Angel@yandex.ru>
> Cc: dmitry@gutov.dev, emacs-devel@gnu.org
> Date: Wed, 03 Apr 2024 13:40:59 +0300
> 
> 1: Emacs does not fully support it though, because while object files
> and configs do end up inside `build`, however `*.elc` files are still
> scattered all over the repo. But I hope you get the idea.

The *.elc files are considered part of the source tree (they are in
the release tarball, and are portable, so no need to rebuild them when
changing some configuration options).  Thus, Emacs does "fully support
it".

Re: [PATCH] Project out of sources compilation

[Konstantin Kharlamov]

On Wed, 2024-04-03 at 14:45 +0300, Eli Zaretskii wrote:
> > From: Konstantin Kharlamov <Hi-Angel@yandex.ru>
> > Cc: dmitry@gutov.dev, emacs-devel@gnu.org
> > Date: Wed, 03 Apr 2024 13:40:59 +0300
> > 
> > 1: Emacs does not fully support it though, because while object
> > files
> > and configs do end up inside `build`, however `*.elc` files are
> > still
> > scattered all over the repo. But I hope you get the idea.
> 
> The *.elc files are considered part of the source tree (they are in
> the release tarball, and are portable, so no need to rebuild them
> when
> changing some configuration options).  

They are generated during compilation and are not commited into git. 
That makes them build artifacts.  The fact they're distributed in
release tarballs does not make it less true.  Distributing them in
tarballs is not strictly necessary, exactly because they can be
generated by anyone.

Having .elc files in the source tree results in problems when a user
created a dir `build-feature1` for one feature branch, and a dir
`build-feature2` for another; and then `build-feature1` somehow ends up
having "feature2".  Which is because it's using build artifacts that
are outside `build-feature1`.

In fact, I myself has experienced some problems due to that while
researching `61960` bug and then later making a separate build dir for
something else.  I don't remember details, but it has been something
with the fact that elc files are outside build dir, and then for some
reason Emacs hasn't been regenerating them or something like that (Idk
exactly what was up with them, I didn't dig into it).

Thus, Emacs certainly does not fully support "out of tree" builds.

Re: [PATCH] Project out of sources compilation

[Eli Zaretskii]

> From: Konstantin Kharlamov <Hi-Angel@yandex.ru>
> Cc: rms@gnu.org, spacibba@aol.com, dmitry@gutov.dev, emacs-devel@gnu.org
> Date: Wed, 03 Apr 2024 16:31:30 +0300
> 
> On Wed, 2024-04-03 at 14:45 +0300, Eli Zaretskii wrote:
> > > From: Konstantin Kharlamov <Hi-Angel@yandex.ru>
> > > Cc: dmitry@gutov.dev, emacs-devel@gnu.org
> > > Date: Wed, 03 Apr 2024 13:40:59 +0300
> > > 
> > > 1: Emacs does not fully support it though, because while object
> > > files
> > > and configs do end up inside `build`, however `*.elc` files are
> > > still
> > > scattered all over the repo. But I hope you get the idea.
> > 
> > The *.elc files are considered part of the source tree (they are in
> > the release tarball, and are portable, so no need to rebuild them
> > when
> > changing some configuration options).  
> 
> They are generated during compilation and are not commited into git. 
> That makes them build artifacts.

How is this relevant to whether or not Emacs supports out of tree
builds?  Building from Git is a special kind of activity; it has its
own INSTALL file and _must_ build some of the files in the source
tree.  Are you aware of many projects whose build from Git can be done
without first populating the source tree with some files?  Heck, you
don't even have the configure script when you clone the repository,
and out-of-tree build works by invoking the configure script from a
directory _other_ than the one where the script lives.  Right?

> The fact they're distributed in release tarballs does not make it
> less true.  Distributing them in tarballs is not strictly necessary,
> exactly because they can be generated by anyone.

Not true.

> Having .elc files in the source tree results in problems when a user
> created a dir `build-feature1` for one feature branch, and a dir
> `build-feature2` for another; and then `build-feature1` somehow ends up
> having "feature2".  Which is because it's using build artifacts that
> are outside `build-feature1`.

Again, not relevant to the aspect on which I commented.

> Thus, Emacs certainly does not fully support "out of tree" builds.

It does.  You are just talking about a very different kind of "out of
tree build".

Re: [PATCH] Project out of sources compilation

[Konstantin Kharlamov]

On Wed, 2024-04-03 at 17:11 +0300, Eli Zaretskii wrote:
> > From: Konstantin Kharlamov <Hi-Angel@yandex.ru>
> > Cc: rms@gnu.org, spacibba@aol.com, dmitry@gutov.dev,
> > emacs-devel@gnu.org
> > Date: Wed, 03 Apr 2024 16:31:30 +0300
> > 
> > On Wed, 2024-04-03 at 14:45 +0300, Eli Zaretskii wrote:
> > > > From: Konstantin Kharlamov <Hi-Angel@yandex.ru>
> > > > Cc: dmitry@gutov.dev, emacs-devel@gnu.org
> > > > Date: Wed, 03 Apr 2024 13:40:59 +0300
> > > > 
> > > > 1: Emacs does not fully support it though, because while object
> > > > files
> > > > and configs do end up inside `build`, however `*.elc` files are
> > > > still
> > > > scattered all over the repo. But I hope you get the idea.
> > > 
> > > The *.elc files are considered part of the source tree (they are
> > > in
> > > the release tarball, and are portable, so no need to rebuild them
> > > when
> > > changing some configuration options).  
> > 
> > They are generated during compilation and are not commited into
> > git. 
> > That makes them build artifacts.
> 
> How is this relevant to whether or not Emacs supports out of tree
> builds? 

Directly: the point of "out of tree build" is encapsulating build
artifacts in the build dir.  I'm pointing out that Emacs has build
artifacts being put outside build dir, hence the "out of tree build" 
workflow is not working as expected.

>  Building from Git is a special kind of activity; it has its
> own INSTALL file and _must_ build some of the files in the source
> tree.  Are you aware of many projects whose build from Git can be
> done
> without first populating the source tree with some files?

Most existing active projects 😊  Most that were using autotools/make
has migrated to Meson.  That includes active freedesktop projects
(Mesa, Xorg, libinput, Pulseaudio, etc), Gnome projects, i3wm, Compton
after it became Picom, all active Enlightenment projects…  The list
goes on and on.  Meson drastically reduces maintainance burden but it's
a separate story.  What matters is that Meson by design works with out
of tree builds.  It is possible to hack in generating files in the
tree, but it's frowned upon for many reasons.

Then you have CMake projects, which I think has always supported out of
tree build without creating any config files whatsoever outside?
Offhand that includes all KDE projects.

Lastly, you have autotools projects where `configure` file is not
generated 😊 `fio` is one example (although admittedly I didn't test
out of tree build in there, but clearly if you have `configure` as part
of the codebase, then you don't have to generate it in the tree).

>   Heck, you
> don't even have the configure script when you clone the repository,
> and out-of-tree build works by invoking the configure script from a
> directory _other_ than the one where the script lives.  Right?

Good point.  That's yet another reason "out of tree build" workflow
isn't fully supported.  I just didn't think about it because didn't
have problems with that, but you are correct here.

> > The fact they're distributed in release tarballs does not make it
> > less true.  Distributing them in tarballs is not strictly
> > necessary,
> > exactly because they can be generated by anyone.
> 
> Not true.

Well, admittedly I didn't try building from release tarball.  But if it
lacks some code to re-create `.elc` files, that's a bug.  I can explain
why if you're interested.

> > Having .elc files in the source tree results in problems when a
> > user
> > created a dir `build-feature1` for one feature branch, and a dir
> > `build-feature2` for another; and then `build-feature1` somehow
> > ends up
> > having "feature2".  Which is because it's using build artifacts
> > that
> > are outside `build-feature1`.
> 
> Again, not relevant to the aspect on which I commented.

Fair enough.  I was just pointing out why "out of tree builds" should
work the way they don't in Emacs right now.  But yeah, if you already
understand that, I guess my explanation on that is unnecessary.

> > Thus, Emacs certainly does not fully support "out of tree" builds.
> 
> It does.  You are just talking about a very different kind of "out of
> tree build".

Well… There does not exist strict definition for "out of tree build"
that I can point you to, so that risks becoming a philosophical debate.

To understand what that means you need to understand what it solves. 
Which you seem to understand already per the previous paragraph, so I
can only add that "partial out of tree build" still has problems that
"full out of tree build" solves.

So there is only one kind of "out of tree build", the "full" one. The
"partial" one should be explicitly specified as being only partial,
whereas saying "out of tree build" without any prefix implies it is a
"full" one.

I hope now you understand why I'm saying Emacs doesn't fully support it
😊

Re: [PATCH] Project out of sources compilation

[Eli Zaretskii]

> From: Konstantin Kharlamov <Hi-Angel@yandex.ru>
> Cc: rms@gnu.org, spacibba@aol.com, dmitry@gutov.dev, emacs-devel@gnu.org
> Date: Wed, 03 Apr 2024 18:00:28 +0300
> 
> I hope now you understand why I'm saying Emacs doesn't fully support it
> 😊

I don't.  I think you are plainly mistaken.

Re: [PATCH] Project out of sources compilation

[Konstantin Kharlamov]

On Wed, 2024-04-03 at 18:47 +0300, Eli Zaretskii wrote:
> > From: Konstantin Kharlamov <Hi-Angel@yandex.ru>
> > Cc: rms@gnu.org, spacibba@aol.com, dmitry@gutov.dev,
> > emacs-devel@gnu.org
> > Date: Wed, 03 Apr 2024 18:00:28 +0300
> >
> > I hope now you understand why I'm saying Emacs doesn't fully
> > support it
> > 😊
>
> I don't.  I think you are plainly mistaken.

Sure, that's a fair point, because if I consider something simple, that doesn't
necessarily mean other people have the knowledge to think similarly.  So to explain
why I was saying that here's some points from my previous email that clarify that:

The point of "out of tree build" is encapsulating build artifacts in the build dir.
Emacs has some build artifacts outside build dir, hence the "out of tree build"
workflow is not working as expected.

To understand what "out of tree build" means you need to understand what it solves.
Which you seem to understand already per prior discussion (but I may be mistaken
here, just tell me if that's so), so let me just add that "partial out of tree build"
still has problems that "full out of tree build" solves.

So there is only one kind of "out of tree build", the "full" one.  The "partial" one
should be explicitly specified as such, whereas saying "out of tree build" without
any prefix implies it is a "full" one.

I hope now you understand why I'm saying Emacs doesn't fully support it 😊

Re: [PATCH] Project out of sources compilation

[Eli Zaretskii]

> From: Konstantin Kharlamov <Hi-Angel@yandex.ru>
> Cc: rms@gnu.org, spacibba@aol.com, dmitry@gutov.dev, emacs-devel@gnu.org
> Date: Wed, 03 Apr 2024 20:27:09 +0300
> 
> I hope now you understand why I'm saying Emacs doesn't fully support it 😊

I still think you are mistaken.  What you call "out of tree build" is
not what is generally meant by that.  Bootstrapping a repository
invariably causes some "build artifacts" to be deposed in the source
tree, and the more the project makes a point of not having generated
files in the repository, the more artifacts are needed in the source
tree before you can start building out of tree.  And that is
perfectly okay, nothing wrong about that.

Re: [PATCH] Project out of sources compilation

[Konstantin Kharlamov]

On Wed, 2024-04-03 at 21:22 +0300, Eli Zaretskii wrote:
> > From: Konstantin Kharlamov <Hi-Angel@yandex.ru>
> > Cc: rms@gnu.org, spacibba@aol.com, dmitry@gutov.dev,
> > emacs-devel@gnu.org
> > Date: Wed, 03 Apr 2024 20:27:09 +0300
> > 
> > I hope now you understand why I'm saying Emacs doesn't fully
> > support it 😊
> 
> I still think you are mistaken.  What you call "out of tree build" is
> not what is generally meant by that.

Okay, what do you consider meant by that?

> Bootstrapping a repository
> invariably causes some "build artifacts" to be deposed in the source
> tree

Sure, that's unfortunately how autotools work.  On the upside though,
it is rare for these files to cause problems.  Which can't be said
about .elc ones.

> , and the more the project makes a point of not having generated
> files in the repository, the more artifacts are needed in the source
> tree before you can start building out of tree.  

Citation needed 😄  You asked me previously for examples of projects
that do not generate any files in the tree, I provided you with that 😊
I can certainly assure you that you don't necessarily have to generate
files "in tree".  I worked with many different projects and distinct
build systems, so I have a lot of experience in that regard.  Do you
have in mind some specific case that you see a problem with?

> And that is
> perfectly okay, nothing wrong about that.

There are problems with generating code in tree, some of which I did
show to you previously.  Unless I misunderstand what you mean by that.

Re: [PATCH] Project out of sources compilation

[Ergus]

Hi Eli:

I won't come into this discussion but:

My basic concept of OOSC (probably mistaken) is that the sources of the
project are those required to reproduce a program, because otherwise it
is not possible or are too difficult/expensive to reproduce.

If I change project.el, then project.elc is not valid anymore and needs
to be regenerated... so, project.el is the source and the elc is a
target (not necessarily a final one)

In the same line, when we do out of sources compilation it may be
possible to remove the `build` directory and have the project sources
exactly as downloaded from sources. Without any other clean
needed...

Also in same line, for example, when using GIT_NAMESPACE the project.el
will change with the namespace, but the elc won't (as not tracked by
git), so GIT_NAMESPACE becomes inconsistent and requires a regeneration
of .elc, so, not a source, but a target

Finally when executing from build1 it may be possible to access the
version of project.elc used when the project was compiled in that
directory... independently of if I changed project.el latter and
compiled in build2. Latter actions in source + compilation in build2
should not affect the consistency of build1.



On Wed, Apr 03, 2024 at 10:08:16PM +0300, Konstantin Kharlamov wrote:
>On Wed, 2024-04-03 at 21:22 +0300, Eli Zaretskii wrote:
>> > From: Konstantin Kharlamov <Hi-Angel@yandex.ru>
>> > Cc: rms@gnu.org, spacibba@aol.com, dmitry@gutov.dev,
>> > emacs-devel@gnu.org
>> > Date: Wed, 03 Apr 2024 20:27:09 +0300
>> >
>> > I hope now you understand why I'm saying Emacs doesn't fully
>> > support it 😊
>>
>> I still think you are mistaken.  What you call "out of tree build" is
>> not what is generally meant by that.
>
>Okay, what do you consider meant by that?
>
>> Bootstrapping a repository
>> invariably causes some "build artifacts" to be deposed in the source
>> tree
>
>Sure, that's unfortunately how autotools work.  On the upside though,
>it is rare for these files to cause problems.  Which can't be said
>about .elc ones.
>
>> , and the more the project makes a point of not having generated
>> files in the repository, the more artifacts are needed in the source
>> tree before you can start building out of tree.  
>
>Citation needed 😄  You asked me previously for examples of projects
>that do not generate any files in the tree, I provided you with that 😊
>I can certainly assure you that you don't necessarily have to generate
>files "in tree".  I worked with many different projects and distinct
>build systems, so I have a lot of experience in that regard.  Do you
>have in mind some specific case that you see a problem with?
>
>> And that is
>> perfectly okay, nothing wrong about that.
>
>There are problems with generating code in tree, some of which I did
>show to you previously.  Unless I misunderstand what you mean by that.

Re: [PATCH] Project out of sources compilation

[Eli Zaretskii]

> Date: Wed, 3 Apr 2024 22:12:44 +0200
> From: Ergus <spacibba@aol.com>
> Cc: Eli Zaretskii <eliz@gnu.org>, rms@gnu.org, dmitry@gutov.dev, 
> 	emacs-devel@gnu.org
> 
> My basic concept of OOSC (probably mistaken) is that the sources of the
> project are those required to reproduce a program, because otherwise it
> is not possible or are too difficult/expensive to reproduce.
> 
> If I change project.el, then project.elc is not valid anymore and needs
> to be regenerated... so, project.el is the source and the elc is a
> target (not necessarily a final one)

The same is true for a change in configure.ac: the configure script is
no longer valid after that, and needs to be rebuilt.

> In the same line, when we do out of sources compilation it may be
> possible to remove the `build` directory and have the project sources
> exactly as downloaded from sources. Without any other clean
> needed...

That's not how the GNU build infrastructure works.  You have the
distclean target in Makefile to clean up the tree for the new build.

> Also in same line, for example, when using GIT_NAMESPACE the project.el
> will change with the namespace, but the elc won't (as not tracked by
> git), so GIT_NAMESPACE becomes inconsistent and requires a regeneration
> of .elc, so, not a source, but a target

That is a problem with project.el, I think, not with how Emacs is
built.  In every project, when you bootstrap it, i.e. build from the
VCS repository, there are some generated files that are part of the
source tree.

> Finally when executing from build1 it may be possible to access the
> version of project.elc used when the project was compiled in that
> directory... independently of if I changed project.el latter and
> compiled in build2. Latter actions in source + compilation in build2
> should not affect the consistency of build1.

You are talking about modifying project.el while it is being used by a
running Emacs session?  In that case, all bets are off, I think.

Re: [PATCH] Project out of sources compilation

[Ergus]

Hi Eli

On April 4, 2024 7:26:35 AM GMT+02:00, Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Wed, 3 Apr 2024 22:12:44 +0200
>> From: Ergus <spacibba@aol.com>
>> Cc: Eli Zaretskii <eliz@gnu.org>, rms@gnu.org, dmitry@gutov.dev, 
>> 	emacs-devel@gnu.org
>> 
>> My basic concept of OOSC (probably mistaken) is that the sources of the
>> project are those required to reproduce a program, because otherwise it
>> is not possible or are too difficult/expensive to reproduce.
>> 
>> If I change project.el, then project.elc is not valid anymore and needs
>> to be regenerated... so, project.el is the source and the elc is a
>> target (not necessarily a final one)
>
>The same is true for a change in configure.ac: the configure script is
>no longer valid after that, and needs to be rebuilt.
>
And that's why the newer build systems generate the equivalent to the configure also out of sources. Or use the equivalent to configure.ac directly as input, generating everything out of sources.


>> In the same line, when we do out of sources compilation it may be
>> possible to remove the `build` directory and have the project sources
>> exactly as downloaded from sources. Without any other clean
>> needed...
>
>That's not how the GNU build infrastructure works.  You have the
>distclean target in Makefile to clean up the tree for the new build.
>
Makefile does not support oosc at all. Actually CMake generates the makefiles outside sources.


>> Also in same line, for example, when using GIT_NAMESPACE the project.el
>> will change with the namespace, but the elc won't (as not tracked by
>> git), so GIT_NAMESPACE becomes inconsistent and requires a regeneration
>> of .elc, so, not a source, but a target
>
>That is a problem with project.el, I think, not with how Emacs is
>built.  In every project, when you bootstrap it, i.e. build from the
>VCS repository, there are some generated files that are part of the
>source tree.
>
Indeed, CMake and actually most of newer autotools projects prefer to generate those files out of sources too.

>> Finally when executing from build1 it may be possible to access the
>> version of project.elc used when the project was compiled in that
>> directory... independently of if I changed project.el latter and
>> compiled in build2. Latter actions in source + compilation in build2
>> should not affect the consistency of build1.
>
>You are talking about modifying project.el while it is being used by a
>running Emacs session?  In that case, all bets are off, I think.

No I mean having two emacs builds in different subdirectories. For example one compiled before  certain change and the other after 




-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [PATCH] Project out of sources compilation

[Eli Zaretskii]

> Date: Thu, 04 Apr 2024 11:59:29 +0200
> From: Ergus <spacibba@aol.com>
> CC: Hi-Angel@yandex.ru, rms@gnu.org, dmitry@gutov.dev, emacs-devel@gnu.org
> 
> >> Finally when executing from build1 it may be possible to access the
> >> version of project.elc used when the project was compiled in that
> >> directory... independently of if I changed project.el latter and
> >> compiled in build2. Latter actions in source + compilation in build2
> >> should not affect the consistency of build1.
> >
> >You are talking about modifying project.el while it is being used by a
> >running Emacs session?  In that case, all bets are off, I think.
> 
> No I mean having two emacs builds in different subdirectories. For example one compiled before  certain change and the other after 

You are saying that Emacs can load project.elc from a wrong directory?
Any reproducible recipe for that?

Re: [PATCH] Project out of sources compilation

[Ergus]

On Thu, Apr 04, 2024 at 02:59:27PM +0300, Eli Zaretskii wrote:
>> Date: Thu, 04 Apr 2024 11:59:29 +0200
>> From: Ergus <spacibba@aol.com>
>> CC: Hi-Angel@yandex.ru, rms@gnu.org, dmitry@gutov.dev, emacs-devel@gnu.org
>>
>> >> Finally when executing from build1 it may be possible to access the
>> >> version of project.elc used when the project was compiled in that
>> >> directory... independently of if I changed project.el latter and
>> >> compiled in build2. Latter actions in source + compilation in build2
>> >> should not affect the consistency of build1.
>> >
>> >You are talking about modifying project.el while it is being used by a
>> >running Emacs session?  In that case, all bets are off, I think.
>>
>> No I mean having two emacs builds in different subdirectories. For example one compiled before  certain change and the other after
>
>You are saying that Emacs can load project.elc from a wrong directory?
>Any reproducible recipe for that?
>
No Eli:

That's not what I mean.

Ex:

mkdir build1 && cd build1 && ../configure ... && make
cd ..

edit project.el
mkdir build2 && cd build2 && ../configure ... && make
cd ..

build1/src/emacs -Q

This will load the new elc, not the one used when it was compiled in
spite of we have not recompiled in that build dir.

Re: [PATCH] Project out of sources compilation

[Eli Zaretskii]

> Date: Thu, 4 Apr 2024 14:34:16 +0200
> From: Ergus <spacibba@aol.com>
> Cc: Hi-Angel@yandex.ru, rms@gnu.org, dmitry@gutov.dev, emacs-devel@gnu.org
> 
> On Thu, Apr 04, 2024 at 02:59:27PM +0300, Eli Zaretskii wrote:
> >
> >You are saying that Emacs can load project.elc from a wrong directory?
> >Any reproducible recipe for that?
> >
> No Eli:
> 
> That's not what I mean.

Actually, yes, that's what you meant.

> mkdir build1 && cd build1 && ../configure ... && make
> cd ..
> 
> edit project.el
> mkdir build2 && cd build2 && ../configure ... && make
> cd ..
> 
> build1/src/emacs -Q
> 
> This will load the new elc, not the one used when it was compiled in
> spite of we have not recompiled in that build dir.

AFAIU, it doesn't load the new project.elc from build2, it loads
project.el (i.e. uncompiled Lisp) from the source tree, where you
modified project.el.

The reason is that Emacs's support for "running uininstalled" assumes
the Lisp files are in the source tree.  IOW, running uninstalled from
a build directory outside of the source tree does not take the *.elc
files from that build directory, as I think you can verify if you look
at the value of load-path.

Re: [PATCH] Project out of sources compilation

[Ergus]

On Thu, Apr 04, 2024 at 04:02:50PM +0300, Eli Zaretskii wrote:
>> Date: Thu, 4 Apr 2024 14:34:16 +0200
>> From: Ergus <spacibba@aol.com>
>> Cc: Hi-Angel@yandex.ru, rms@gnu.org, dmitry@gutov.dev, emacs-devel@gnu.org
>>
>> On Thu, Apr 04, 2024 at 02:59:27PM +0300, Eli Zaretskii wrote:
>> >
>> >You are saying that Emacs can load project.elc from a wrong directory?
>> >Any reproducible recipe for that?
>> >
>> No Eli:
>>
>> That's not what I mean.
>
>Actually, yes, that's what you meant.
>
>> mkdir build1 && cd build1 && ../configure ... && make
>> cd ..
>>
>> edit project.el
>> mkdir build2 && cd build2 && ../configure ... && make
>> cd ..
>>
>> build1/src/emacs -Q
>>
>> This will load the new elc, not the one used when it was compiled in
>> spite of we have not recompiled in that build dir.
>
>AFAIU, it doesn't load the new project.elc from build2, it loads
>project.el (i.e. uncompiled Lisp) from the source tree, where you
>modified project.el.
>
>The reason is that Emacs's support for "running uininstalled" assumes
>the Lisp files are in the source tree.  IOW, running uninstalled from
>a build directory outside of the source tree does not take the *.elc
>files from that build directory, as I think you can verify if you look
>at the value of load-path.

Exactly, so this is not OOSC and does not behave as expected in
OOSC. Which is what Konstantin pointed out since the beginning.

Re: [PATCH] Project out of sources compilation

[Eli Zaretskii]

> Date: Thu, 4 Apr 2024 16:27:34 +0200
> From: Ergus <spacibba@aol.com>
> Cc: Hi-Angel@yandex.ru, rms@gnu.org, dmitry@gutov.dev, emacs-devel@gnu.org
> 
> >The reason is that Emacs's support for "running uininstalled" assumes
> >the Lisp files are in the source tree.  IOW, running uninstalled from
> >a build directory outside of the source tree does not take the *.elc
> >files from that build directory, as I think you can verify if you look
> >at the value of load-path.
> 
> Exactly, so this is not OOSC and does not behave as expected in
> OOSC.

No, it isn't.  Not many programs at all support running uninstalled.
Emacs does, but only if you build inside the source.  This is
unrelated to the possibility of out-of-source build, where you are
supposed to say "make install" after you build.

> Which is what Konstantin pointed out since the beginning.

No, he didn't.

Re: [PATCH] Project out of sources compilation

[Ergus]

On April 4, 2024 4:41:58 PM GMT+02:00, Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Thu, 4 Apr 2024 16:27:34 +0200
>> From: Ergus <spacibba@aol.com>
>> Cc: Hi-Angel@yandex.ru, rms@gnu.org, dmitry@gutov.dev, emacs-devel@gnu.org
>> 
>> >The reason is that Emacs's support for "running uininstalled" assumes
>> >the Lisp files are in the source tree.  IOW, running uninstalled from
>> >a build directory outside of the source tree does not take the *.elc
>> >files from that build directory, as I think you can verify if you look
>> >at the value of load-path.
>> 
>> Exactly, so this is not OOSC and does not behave as expected in
>> OOSC.
>
>No, it isn't.  Not many programs at all support running uninstalled.

All CMake and Meson programs do it by default. Mainly because they also provide test features  where the executables are supposed to be tested before calling install or pack.

>Emacs does, but only if you build inside the source.  This is
>unrelated to the possibility of out-of-source build, where you are
>supposed to say "make install" after you build.
>
>> Which is what Konstantin pointed out since the beginning.
>
>No, he didn't.
>

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [PATCH] Project out of sources compilation

[Eli Zaretskii]

> Date: Thu, 04 Apr 2024 20:15:56 +0200
> From: Ergus <spacibba@aol.com>
> CC: Hi-Angel@yandex.ru, rms@gnu.org, dmitry@gutov.dev
> 
> >> Exactly, so this is not OOSC and does not behave as expected in
> >> OOSC.
> >
> >No, it isn't.  Not many programs at all support running uninstalled.
> 
> All CMake and Meson programs do it by default. Mainly because they also provide test features  where the executables are supposed to be tested before calling install or pack.

I don't see how CMake or Meson can make a program run uninstalled,
unless the program already provides the necessary features for doing
so.  Imagine a program that needs to read some data files, which it
expects to find in a directory under /usr/share whose absolute file
name is hard-coded into it, as determined from --prefix at configure
time.  Such a program cannot run uninstalled, because the files in
/usr/share are not yet installed.

Emacs needs to know where to find the Lisp files it loads on demand.
If those files are in the place configured via --prefix, they can be
found easily, but if Emacs runs uninstalled, the Lisp files are not
there, and Emacs needs to find them somehow.  Unless someone writes
code to look for them in the right places, there's no way CMake or
Meson or any other general-purpose tool could fix that.

Re: [PATCH] Project out of sources compilation

[Konstantin Kharlamov]

On Thu, 2024-04-04 at 17:41 +0300, Eli Zaretskii wrote:
> > Date: Thu, 4 Apr 2024 16:27:34 +0200
> > From: Ergus <spacibba@aol.com>
> > Cc: Hi-Angel@yandex.ru, rms@gnu.org, dmitry@gutov.dev,
> > emacs-devel@gnu.org
> > 
> > > The reason is that Emacs's support for "running uininstalled"
> > > assumes
> > > the Lisp files are in the source tree.  IOW, running uninstalled
> > > from
> > > a build directory outside of the source tree does not take the
> > > *.elc
> > > files from that build directory, as I think you can verify if you
> > > look
> > > at the value of load-path.
> > 
> > Exactly, so this is not OOSC and does not behave as expected in
> > OOSC.
> 
> No, it isn't.  Not many programs at all support running uninstalled.
> Emacs does, but only if you build inside the source.  This is
> unrelated to the possibility of out-of-source build, where you are
> supposed to say "make install" after you build.

Indeed running uninstalled is kind of orthogonal to OOSC and may or may
not be supported by a program.  I think this discussion might be
confusing two orthogonal things: α) `.elc` files being put in-tree, and
β) Emacs requiring for them to be put in-tree to run "uninstalled".

For β: I'm sure it's possible to modify Emacs to load `.elc` from the
build dir rather than from the tree.

For α: the lone fact that build artifacts are being put in-tree means
OOSC is not working as expected.

I see what you mean by saying that `configure` and similar files can't
be put into the build dir because that's how autotools work.  Okay. 
But I think this whole discussion about the terminology is becoming not
very useful.  Can we please agree that generating .elc files in the
build dir rather than in tree allows to avoid a number of problems? 
Please note that this has nothing to do with `.elc` files being put in
tarballs, because for that to work you could just build "in-tree" as
usual.  Putting `.elc` files into build dir (which may be in-tree or
out) and allowing Emacs to take those into account is just
simplification in how things work.

Re: [PATCH] Project out of sources compilation

[Eli Zaretskii]

> From: Konstantin Kharlamov <Hi-Angel@yandex.ru>
> Cc: rms@gnu.org, dmitry@gutov.dev, emacs-devel@gnu.org
> Date: Thu, 04 Apr 2024 23:16:29 +0300
> 
> Can we please agree that generating .elc files in the
> build dir rather than in tree allows to avoid a number of problems? 
> Please note that this has nothing to do with `.elc` files being put in
> tarballs, because for that to work you could just build "in-tree" as
> usual.  Putting `.elc` files into build dir (which may be in-tree or
> out) and allowing Emacs to take those into account is just
> simplification in how things work.

As I wrote up-thread, I have no objections for adding this to Emacs,
but it is not as trivial as it seems, because Emacs needs more files
than just *.elc to run, and some of those other files are in the
source tree when Emacs runs uninstalled (example: the images in
etc/images).  So adding this mode of running Emacs would need some
changes to the C startup code, not just to Makefile's.  Also, if you
edit *.el files in the source directory, the *.eln files produced from
the *.el files before the edits will no longer be loaded by Emacs
(because they don't match the corresponding *.el files); if we want to
fix that as well, we'd need much more serious changes.

Re: [PATCH] Project out of sources compilation

[Eli Zaretskii]

> From: Konstantin Kharlamov <Hi-Angel@yandex.ru>
> Cc: rms@gnu.org, spacibba@aol.com, dmitry@gutov.dev, emacs-devel@gnu.org
> Date: Wed, 03 Apr 2024 22:08:16 +0300
> 
> On Wed, 2024-04-03 at 21:22 +0300, Eli Zaretskii wrote:
> > > From: Konstantin Kharlamov <Hi-Angel@yandex.ru>
> > > Cc: rms@gnu.org, spacibba@aol.com, dmitry@gutov.dev,
> > > emacs-devel@gnu.org
> > > Date: Wed, 03 Apr 2024 20:27:09 +0300
> > > 
> > > I hope now you understand why I'm saying Emacs doesn't fully
> > > support it 😊
> > 
> > I still think you are mistaken.  What you call "out of tree build" is
> > not what is generally meant by that.
> 
> Okay, what do you consider meant by that?

In general, a build outside of the tree where a release tarball has
been unpacked.  At the very least, a build _after_ autogen.sh (or a
similar "bootstrap" script) was run.

Note that the rules which produce *.elc files do deposit them in the
build tree, not in the source tree, btw.  AFAIR, the only Lisp files
produced by a build from Git that are deposited in the source tree are
those built by admin/grammars/Makefile.  If someone wants to modify
that Makefile to produce the *.el files in the build tree, I won't
object, but rejecting the Emacs out-of-tree build procedure because we
currently put them in the source tree is absurd, IMNSHO, since *.el
files are certainly portable and not configuration- or
platform-dependent.  Putting platform-independent generated files in
the source tree has the advantage of making the subsequent builds
faster, sometimes significantly faster.

> > Bootstrapping a repository
> > invariably causes some "build artifacts" to be deposed in the source
> > tree
> 
> Sure, that's unfortunately how autotools work.

See above: not just autotools.  And anyway, as long as the Emacs build
uses autotools, we can only support out-of-tree builds in the style
supported by autotools.

> On the upside though, it is rare for these files to cause problems.
> Which can't be said about .elc ones.

See above: the *.elc files are not relevant here.

> > , and the more the project makes a point of not having generated
> > files in the repository, the more artifacts are needed in the source
> > tree before you can start building out of tree.  
> 
> Citation needed 😄  You asked me previously for examples of projects
> that do not generate any files in the tree, I provided you with that 😊
> I can certainly assure you that you don't necessarily have to generate
> files "in tree".  I worked with many different projects and distinct
> build systems, so I have a lot of experience in that regard.  Do you
> have in mind some specific case that you see a problem with?

Yes, look at Gawk, for example: they have many generated files in Git.
Even Info files are in Git.

Another example is GDB: they store the configure script in Git.

> > And that is
> > perfectly okay, nothing wrong about that.
> 
> There are problems with generating code in tree, some of which I did
> show to you previously.

There are problems everywhere, but those are quite easy to deal with.
Otherwise you wouldn't be seeing projects doing that, and doing that
on purpose.

Re: [PATCH] Project out of sources compilation

[Konstantin Kharlamov]

Wow. Please, offense is unwelcome.  Eli is a great man who maintains
the project and writes code. Other people come and go, Eli stays.  I'm
very thankful to him for doing that job.  Disagreement happens, that's
okay.  No need to call names.