Re: [PATCH] Project out of sources compilation

[Mohsin Kaleem <mohkale@kisara.moe>]

Ergus <spacibba@aol.com> writes:

Howdie howdie,

I'm the maintainer of projection. Looks like it's a similar external
package to what we're trying to formalize in Emacs core project.el .

I'm just going to highlight 2 things which I think would help you Ergus.
Ninja multi-config generators [1] which basically lets you configure a
project for Debug, Release, ASAN, UBSAN, etc. simultaneously. And CMake
presets [2] which is a standard for loading common configuration in CMake
like build directories but also compiler flags, environment variables,
etc.

[1]: https://cmake.org/cmake/help/latest/generator/Ninja%20Multi-Config.html
[2]: https://cmake.org/cmake/help/latest/manual/cmake-presets.7.html

I'll add my 2-cents on some of the discussed points.

Firstly, I disagree with the goal of making the build directory option
customizable in project.el itself. Most build tools I've encountered
(including all supported by projection) take an extra flag to point to
the build directory and I find this a nicer UX because it's easily
inspectable and editable with `C-u M-x recompile`. I noticed projectile
supported this feature of "run somewhere else" as well but it was only
for one project type "meson" and even that has this option to move
directories so I don't really see the value of complicating the
interface to support it. Another pain point with a generalised
"build-directory" option is that it's not generic for all possible
commands. Configuration often runs before the build directory exists
(because it creates it) so then you need an exemption for one command
type but not another. I think the project-root as a standard location
for commands to run is a good common default and changing it unless
absolutely necessary is un-needed :-).

Secondly I agree with Dmitry about defining an alternate orthoganal API
to project.el for this. It probably belongs more in a project-types.el
package. The motivation being that project.el is for more bare-bones
mechanics, things like "am I in a project", "where is the root of the
project", "show me all the files in this project". Project types is more
"how do I build this project", "how do I test it", "what artifacts does it
produce". The former is a set of functionalities which is relatively
common and static across all projects, the latter is very specific to
what kind of a project we're dealing with and how you interface with it.
For example python projects probably don't have a configure step or
build directory, node projects may treat "npm install" as a configure
step but omit building, CMake projects support all steps through to
packaging and installing. Things don't generalise well here.

I think Ergus you've actually touched on a fun corner case. You've got a
project with sub-projects and define a regex in a marker file to detect
this. I'll admit I haven't touched on this before, I know you can have a
CMake project with different sub-directories being considered their own
projects and buildable independently, but I've never seen this practice
in my day-to-day work. I'd be inclined to treat this as a project.el
feature, something akin to git sub-modules. My only concern with it as a
feature is its not trivial to check like git submodules. Basically every
directory in a CMake project can have a CMakeLists.txt file and
recursively searching upwards for one that might coincidentally also be
a project root marker isn't ideal :-(.

While we're discussing interfaces for this project-type abstraction I'll
mention what I did for projection. In projection a project-type is an
eioio class. It has some slots like :predicate to check if the current
project matches that type and :build, :test, etc. for various project
specific commands that can be run. The list of all actively checked-for
project types is just a flat hook. I opted for this for 2 reasons:

1. It's easy to modify. Every project-type can be edited directly you
can deregistered one from the monitored set of types by just removing it
from the collection.
2. It enforces a general ordering concept which is important for
determining a primary project type. I think moving the properties into a
generic function though would just decouple them and make it harder to
reason about and modify them.

In regards to interactivity, I outsourced a lot to another package I
maintain called compile-multi [3]. At its core its just a menu for
predicates and compile commands. When predicate X, you have the option
to choose command Y. I've plugged it into projection so that through it
you can see project type specific options. For example in a CMake
project this menu shows you all compilation targets and all CTest
targets. Running a selected target will run the compile command that
builds that target. If you want to permanently override one that runs
with the standard projection-commands-build-project or
projection-commands-test-project interactive commands (this is the same
as setting what you're wanting to build going forward) I recommend using
of embark [4]. What I normally do is run `M-x projection-multi-compile`
narrow to the build or test target I care about and then run `M-x
embark-act p c` (or p t for a test) and next time I run
projection-commands-build-project it'll use that selected command. If I
want to reset this to the project default I can use `M-x
projection-cache-clear` which shows all cached project options and lets
me select one or more to prune. Next time I run any command it'll be
the default for the current project.

[3]: https://github.com/mohkale/compile-multi
[4]: https://github.com/oantolin/embark

I think generalising this latter feature to a common interface will be
tricky. To begin with just knowing a project has build target X doesn't
really tell you how to build X, just that you can. You need to feed that
selection back into the project-type or export it directly as is. I
recently added a feature to list build artifacts (for debugger
integration) and that shows just how much variety there is here.

Lastly, just a general point (sorry for writing so much), I think any
feature for this shouldn't assume only a single project type is valid.
There should be a concept of a primary type just to make commands like
configure or build make sense relative to each other, but nothing stops
a project from matching two types at once and at least IME that's the
more natural state of things. I commonly work on C++ projects but we
also have a package.json in those projects file so we can depend on
prettier and other non-C++ specific linters. The way projection gets
around this is by always matching all defined project types. Users can
add to the list of matched types for the current project and can cycle
the primary project type. Certain interfaces that aren't single project
specific like the aforementioned projection-multi-compile will source
compilation targets from all applicable project types. This provides an
extremely rich interface for interacting with projects (at least IMO).

Let me know if anyone has any thoughts or questions about projection.
I'm not averse to re-aligning across something in Emacs mainline. The
only minor concern is how specific it is to the kinds of projects I work
on and that may not generalise well to others.

-- 
Mohsin Kaleem