bug#70408: 30.0.50; Eglot and Project integration

bug#70408: 30.0.50; Eglot and Project integration

[Ergus via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Hi:

Recently we have been discussing the possibility to improve project.el
in order to recognize/interact a bit more smartly with common modern
build infrastructures (i.e meson and cmake).

These build systems generally are also capable to generate the
compile_commands.json for clangd in the build directory independently of
its location (generally out of sources).

The current extension for project.el is capable to recognize the build
directory to execute project-compile. The approach works well in the
tests and I added a small POC code to modify the
`glot-workspace-configuration' variable on the fly.

This integration of project.el is pretty useful and simplifies the
configurations required to make eglot work a bit more consistently by
detecting the database more accurately.

The only limitation I am facing at the moment with this is that
project.el initializes lazily (when a project-something command is
called) and generally eglot seems to be designed to autostart as a mode
hook.

So the issue has two parts:

1. Is it desirable or are the eglot developers somehow interested in the
integration with project.el? If so, what are the key features of
interest.

2. Do you have some suggestion about how to initialize the eglot server
to properly update the `glot-workspace-configuration' on the fly?

My very primitive proof of concept code:

https://github.com/Ergus/project-multi-mode/blob/822316d82007e1b68c9a8dfcfbe205cb63b4f545/project-multi-mode.el#L194

Best,

bug#70408: 30.0.50; Eglot and Project integration

[Eli Zaretskii]

> Date: Mon, 15 Apr 2024 23:40:27 +0200
> From:  Ergus via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> Recently we have been discussing the possibility to improve project.el
> in order to recognize/interact a bit more smartly with common modern
> build infrastructures (i.e meson and cmake).
> 
> These build systems generally are also capable to generate the
> compile_commands.json for clangd in the build directory independently of
> its location (generally out of sources).
> 
> The current extension for project.el is capable to recognize the build
> directory to execute project-compile. The approach works well in the
> tests and I added a small POC code to modify the
> `glot-workspace-configuration' variable on the fly.
> 
> This integration of project.el is pretty useful and simplifies the
> configurations required to make eglot work a bit more consistently by
> detecting the database more accurately.
> 
> The only limitation I am facing at the moment with this is that
> project.el initializes lazily (when a project-something command is
> called) and generally eglot seems to be designed to autostart as a mode
> hook.
> 
> So the issue has two parts:
> 
> 1. Is it desirable or are the eglot developers somehow interested in the
> integration with project.el? If so, what are the key features of
> interest.
> 
> 2. Do you have some suggestion about how to initialize the eglot server
> to properly update the `glot-workspace-configuration' on the fly?
> 
> My very primitive proof of concept code:
> 
> https://github.com/Ergus/project-multi-mode/blob/822316d82007e1b68c9a8dfcfbe205cb63b4f545/project-multi-mode.el#L194

Thanks.

I think this discussion should include João, so I added him.

bug#70408: 30.0.50; Eglot and Project integration

[João Távora]

On Tue, Apr 16, 2024 at 12:42 PM Eli Zaretskii <eliz@gnu.org> wrote:

> I think this discussion should include João, so I added him.

Alright.

More importantly, I think this discussion should include Dmitry, as this seems
to me a project.el extension.

This discussion should be aware of these half-recent developments
https://github.com/joaotavora/eglot/discussions/1337

In a few words, Eglot user's main gripe with project.el is project.el's
inability to help the user define or designate subprojects within
larger projects.
Eglot has worked around that, and the current work-around is very
effective (though not really well documented beyond that GitHub discussion
forum).

If Ergus's developments change project.el so that this special Eglot code
isn't necessary, that's great IMO.  If they make it so that Eglot now has
to depend on extra package and extra project.el features, that's not so
great.

That's my opinion as current Eglot maintainer.  A capable future
Eglot maintainer may have another opinion and want to steer ship
in a completely different direction.  I'm looking for such a person, btw.

João

bug#70408: 30.0.50; Eglot and Project integration

[Dmitry Gutov]

Hi Joao,

On 16/04/2024 15:33, João Távora wrote:
> On Tue, Apr 16, 2024 at 12:42 PM Eli Zaretskii<eliz@gnu.org>  wrote:
> 
>> I think this discussion should include João, so I added him.
> Alright.
> 
> More importantly, I think this discussion should include Dmitry, as this seems
> to me a project.el extension.
> 
> This discussion should be aware of these half-recent developments
> https://github.com/joaotavora/eglot/discussions/1337
> 
> In a few words, Eglot user's main gripe with project.el is project.el's
> inability to help the user define or designate subprojects within
> larger projects.

IIUC Ergus's request is primarily about a situation where an 
"out-of-tree" build is used. Meaning, the directory for build artefacts 
is not a subdirectory of the project root, but -- apparently -- some 
sibling directory of it (e.g. "../build"). So it's somewhat atypical, 
although I suppose the solution from the link above might work with it too.

This bug is split off from an emacs-devel discussion, where I posted a 
draft solution of mine: 
https://lists.gnu.org/archive/html/emacs-devel/2024-04/msg00279.html

I'm curious for any feedback - like would that be good enough for this 
and related cases, or maybe if someone has an even simpler approach in mind.

bug#70408: 30.0.50; Eglot and Project integration

[Ergus via Bug reports for GNU Emacs, the Swiss army knife of text editors]

On Tue, Apr 16, 2024 at 01:33:31PM +0100, João Távora wrote:
>On Tue, Apr 16, 2024 at 12:42 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
>> I think this discussion should include João, so I added him.
>
>Alright.
>
>More importantly, I think this discussion should include Dmitry, as this seems
>to me a project.el extension.
>
Indeed. Actually Dmitry suggested me to open the bug.

>This discussion should be aware of these half-recent developments
>https://github.com/joaotavora/eglot/discussions/1337
>
I will give it a look.

>In a few words, Eglot user's main gripe with project.el is project.el's
>inability to help the user define or designate subprojects within
>larger projects.
>Eglot has worked around that, and the current work-around is very
>effective (though not really well documented beyond that GitHub discussion
>forum).
>
A bit more details on the workaround and its final state may be useful.


>If Ergus's developments change project.el so that this special Eglot code
>isn't necessary, that's great IMO.  If they make it so that Eglot now has
>to depend on extra package and extra project.el features, that's not so
>great.
>
I would like to avoid inter-dependencies, however, considering that
Eglot and project.el are both in vanilla; maybe we could be a bit
flexible here.

However, I thing that there are some ways to handle that.

So far, what I had in mind was a sort of non-intrusive api (either in
eglot or project.el) that could be used by project.el (and/or a
project.el backend) to provide more accurate information to eglot.

As I said before, the real issue is that project.el is intended to
initialize lazily while eglot is generally already running; so the only
realistic alternatives I see are:

1. Make eglot call a project.el function to detect the build-dir and
compile-commands.json (or equivalent)

2. Make project.el to check if eglot is enabled and call some eglot
function to update the eglot's "build-dir".

 From these two I actually prefer the second one because it may happen
that a project has different build-dirs each of them with a different
compile-commands.json (i.e debug and release) and the user wants to
change build-dir dynamically and potentially inform eglot about the
change.

>That's my opinion as current Eglot maintainer.  A capable future
>Eglot maintainer may have another opinion and want to steer ship
>in a completely different direction.  I'm looking for such a person, btw.
>
>João

bug#70408: 30.0.50; Eglot and Project integration

[João Távora]

On Tue, Apr 16, 2024 at 1:56 PM Dmitry Gutov <dmitry@gutov.dev> wrote:

> IIUC Ergus's request is primarily about a situation where an
> "out-of-tree" build is used. Meaning, the directory for build artefacts
> is not a subdirectory of the project root, but -- apparently -- some
> sibling directory of it (e.g. "../build"). So it's somewhat atypical,
> although I suppose the solution from the link above might work with it too.

Ah, I know about that.  That's where compile-commands.json is generated
by CMake.  But using that './build' as the project root passed via LSP
to clangd
(and likely any other server) will most likely fail: that's not  the
project root
and it doesn't have any versioned source files  (only auto-generated ones).

Because of this, C++ projects usually have sth like:

  ln -sf build/compile_commands.json compile_commands.json

as a build step.

Alternatively, you invoke clangd with `--compile-commands-dir=build`.
I don't think there's anything project.el or Eglot can do or should do
about this case.

> This bug is split off from an emacs-devel discussion, where I posted a
> draft solution of mine:
> https://lists.gnu.org/archive/html/emacs-devel/2024-04/msg00279.html
> I'm curious for any feedback - like would that be good enough for this
> and related cases, or maybe if someone has an even simpler approach in mind.

I'll pass, but wish you luck.  I've stated in the past that I think
project.el should
allow subprojects inside larger projects, and let users of
project-current (direct
or indirect) specify if they're interesting in the innermost,
outermost, or intermediate
projects when searching for projects to act on, via a combination of prefix
arguments, arguments, customized special variables or let-bound special
variables.

João

bug#70408: 30.0.50; Eglot and Project integration

[Ergus via Bug reports for GNU Emacs, the Swiss army knife of text editors]

On Tue, Apr 16, 2024 at 02:51:10PM +0100, João Távora wrote:
>On Tue, Apr 16, 2024 at 1:56 PM Dmitry Gutov <dmitry@gutov.dev> wrote:
>
>> IIUC Ergus's request is primarily about a situation where an
>> "out-of-tree" build is used. Meaning, the directory for build artefacts
>> is not a subdirectory of the project root, but -- apparently -- some
>> sibling directory of it (e.g. "../build"). So it's somewhat atypical,
>> although I suppose the solution from the link above might work with it too.
>
>Ah, I know about that.  That's where compile-commands.json is generated
>by CMake.  But using that './build' as the project root passed via LSP
>to clangd
>(and likely any other server) will most likely fail: that's not  the
>project root
>and it doesn't have any versioned source files  (only auto-generated ones).
>
>Because of this, C++ projects usually have sth like:
>
>  ln -sf build/compile_commands.json compile_commands.json
>
>as a build step.
>
>Alternatively, you invoke clangd with `--compile-commands-dir=build`.
>I don't think there's anything project.el or Eglot can do or should do
>about this case.
>
What my POC workaround does is basically call a hook that updates
`eglot-workspace-configuration` as a root's directory-local variable
the first time project-current is called within a project.


```
(defun project-multi--set-eglot (plist)
   "Set the eglot variables in root's PLIST when possible."
   (when-let* ((compile-dir (project-extra-info (project-current) :compile-dir))
	      (file-exists-p (expand-file-name "compile_commands.json" compile-dir)))

     (let* ((symvars (intern (format "eglot-multi--%s" compile-dir)))
	   (eglot-complete (project-multi--merge-plist ;; merge with new values
			    (bound-and-true-p eglot-workspace-configuration)
			    `(:clangd (:initializationOptions
				       (:compilationDatabasePath ,compile-dir))))))

       ;; set the dir local variables, they will apply automatically to
       ;; all buffers open in the future within the project root
       (dir-locals-set-class-variables
        symvars
        `((nil . ((eglot-workspace-configuration . ,eglot-complete)))))

       (dir-locals-set-directory-class (project-root (project-current)) symvars)

       ;; set the variable manually in all the already opened buffers
       ;; TODO: JAM check if the variable is not already set in the other buffers??
       ;; Probably override only the value instead of replacing the whole variable?
       (mapc (lambda (buffer)
	      (with-current-buffer buffer
		(setq-local eglot-workspace-configuration eglot-complete)))
	    (project-buffers (project-current))))))
```

project-multi--merge-plist is just a hack function to merge the
eglot-workspace-configuration value without overriding the existing
sub-values if already set (in case the user sets some sub-values in the
dir locals then those takes precedence)

Then this restarts eglot

IIUC this is equivalent to call `--compile-commands-dir=build` and at
the moment is working for me.


>> This bug is split off from an emacs-devel discussion, where I posted a
>> draft solution of mine:
>> https://lists.gnu.org/archive/html/emacs-devel/2024-04/msg00279.html
>> I'm curious for any feedback - like would that be good enough for this
>> and related cases, or maybe if someone has an even simpler approach in mind.
>
>I'll pass, but wish you luck.  I've stated in the past that I think
>project.el should
>allow subprojects inside larger projects, and let users of
>project-current (direct
>or indirect) specify if they're interesting in the innermost,
>outermost, or intermediate
>projects when searching for projects to act on, via a combination of prefix
>arguments, arguments, customized special variables or let-bound special
>variables.
>
At the moment I assume the outer most only; which is the simpler one to
setup and implement woth the current project.el support. The main goal I
have is OOSC, not nested projects.

>João

bug#70408: 30.0.50; Eglot and Project integration

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Tue, 16 Apr 2024 13:33:31 +0100
> Cc: Ergus <spacibba@aol.com>, 70408@debbugs.gnu.org
> 
> On Tue, Apr 16, 2024 at 12:42 PM Eli Zaretskii <eliz@gnu.org> wrote:
> 
> > I think this discussion should include João, so I added him.
> 
> Alright.
> 
> More importantly, I think this discussion should include Dmitry, as this seems
> to me a project.el extension.

Of course.  However, Dmitry reads the bug list, so I reckoned he
didn't need this discussion to be forwarded to him.

bug#70408: 30.0.50; Eglot and Project integration

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1278 bytes --]

On Tue, Apr 16, 2024 at 5:02 PM Ergus <spacibba@aol.com> wrote:

> project-multi--merge-plist is just a hack function to merge the
> eglot-workspace-configuration value without overriding the existing
> sub-values if already set (in case the user sets some sub-values in the
> dir locals then those takes precedence)
>
> Then this restarts eglot
>
> IIUC this is equivalent to call `--compile-commands-dir=build` and at
> the moment is working for me.

This seems extremely complicated, but happy it works for you.

If during an eglot session something happens that leads to want to update
the "workspace configuration" for a given session, you can set it and
then call `eglot-signal-didChangeConfiguration` which is part of Eglot's
API. No need to restart.  Also note that eglot-workspace-configuration
can be a function, maybe that's useful to you somehow.

> At the moment I assume the outer most only; which is the simpler one to
> setup and implement woth the current project.el support. The main goal I
> have is OOSC, not nested projects.

OK.  I don't know what OOSC is but if it's somehow related to "out of tree
builds"
I think these are fine, but builds aren't normally not part of a project:
my
.gitignore files ignores them.

João

[-- Attachment #2: Type: text/html, Size: 1557 bytes --]

bug#70408: 30.0.50; Eglot and Project integration

[Ergus via Bug reports for GNU Emacs, the Swiss army knife of text editors]

On Tue, Apr 16, 2024 at 09:30:20PM +0100, João Távora wrote:
>On Tue, Apr 16, 2024 at 5:02 PM Ergus <spacibba@aol.com> wrote:
>
>> project-multi--merge-plist is just a hack function to merge the
>> eglot-workspace-configuration value without overriding the existing
>> sub-values if already set (in case the user sets some sub-values in the
>> dir locals then those takes precedence)
>>
>> Then this restarts eglot
>>
>> IIUC this is equivalent to call `--compile-commands-dir=build` and at
>> the moment is working for me.
>
>This seems extremely complicated, but happy it works for you.
>
The core idea is actually very simple: if project.el detects a
build-dir with compile-commands.json inside then update
eglot-workspace-configuration. The rest is just the api to set directory
local vars (extremely complicated indeed, but that's what it is)

>If during an eglot session something happens that leads to want to update
>the "workspace configuration" for a given session, you can set it and
>then call `eglot-signal-didChangeConfiguration` which is part of Eglot's
>API. No need to restart.  Also note that eglot-workspace-configuration
>can be a function, maybe that's useful to you somehow.
>
This is actually very useful; it is probably the only feature I needed
in the eglot side.

>> At the moment I assume the outer most only; which is the simpler one to
>> setup and implement woth the current project.el support. The main goal I
>> have is OOSC, not nested projects.
>
>OK.  I don't know what OOSC is but if it's somehow related to "out of tree
>builds" 
Out Of Sources Compilation

>I think these are fine, but builds aren't normally not part of a project:
>my
>.gitignore files ignores them.
>
Indeed, but my function in project.el won't.

The code I wrote considers some file-names as root hints (i.e
CMakeLists.txt) and others as build dirs hints (i.e CMakeCache.txt).

My project-find-functions hook searches upwards in the directory trees
for a root hint and in the top most root it searches not recursively for
the build dir hint in the root's sub-directories.

When there are multiple build dirs it asks to the user for which one to
use.

And after that it searches for the compile-commands.json there.

The code also sets a directory-local-var to avoid repeating this search
for all the other files in the future; because I use Tramp extensively
and such searches may slow.

>João

bug#70408: 30.0.50; Eglot and Project integration

[Dmitry Gutov]

On 16/04/2024 16:51, João Távora wrote:
> On Tue, Apr 16, 2024 at 1:56 PM Dmitry Gutov <dmitry@gutov.dev> wrote:
> 
>> IIUC Ergus's request is primarily about a situation where an
>> "out-of-tree" build is used. Meaning, the directory for build artefacts
>> is not a subdirectory of the project root, but -- apparently -- some
>> sibling directory of it (e.g. "../build"). So it's somewhat atypical,
>> although I suppose the solution from the link above might work with it too.
> 
> Ah, I know about that.  That's where compile-commands.json is generated
> by CMake.  But using that './build' as the project root passed via LSP
> to clangd
> (and likely any other server) will most likely fail: that's not  the
> project root
> and it doesn't have any versioned source files  (only auto-generated ones).
> 
> Because of this, C++ projects usually have sth like:
> 
>    ln -sf build/compile_commands.json compile_commands.json
> 
> as a build step.

Would you say it's common across most projects? E.g. generated as a 
build step by CMake in out-of-tree comfigurations.

Or is it something Emacs users tend to add.

What I'm wondering is whether we might be currently disadvantaged 
compared to the users of some other editors, which might have some more 
auto-detection in that area.

> Alternatively, you invoke clangd with `--compile-commands-dir=build`.
> I don't think there's anything project.el or Eglot can do or should do
> about this case.
> 
>> This bug is split off from an emacs-devel discussion, where I posted a
>> draft solution of mine:
>> https://lists.gnu.org/archive/html/emacs-devel/2024-04/msg00279.html
>> I'm curious for any feedback - like would that be good enough for this
>> and related cases, or maybe if someone has an even simpler approach in mind.
> 
> I'll pass, but wish you luck.  I've stated in the past that I think
> project.el should
> allow subprojects inside larger projects, and let users of
> project-current (direct
> or indirect) specify if they're interesting in the innermost,
> outermost, or intermediate
> projects when searching for projects to act on, via a combination of prefix
> arguments, arguments, customized special variables or let-bound special
> variables.

I'm open to continuing this discussion somewhere, but it's unrelated to 
this particular one. Note that, as I explained, it would be easy to 
create a set of commands acting on "super" projects, but that's probably 
not what you're looking for.

bug#70408: 30.0.50; Eglot and Project integration

[João Távora]

On Sat, Apr 20, 2024 at 12:28 PM Dmitry Gutov <dmitry@gutov.dev> wrote:

> > Because of this, C++ projects usually have sth like:
> >
> >    ln -sf build/compile_commands.json compile_commands.json
> >
> > as a build step.
>
> Would you say it's common across most projects? E.g. generated as a
> build step by CMake in out-of-tree comfigurations.
> Or is it something Emacs users tend to add.

Pretty common, yes.  I googled for the ln string:

https://pspdfkit.com/blog/2019/visual-studio-code-for-cpp/  VSCode
https://blog.justforlxz.com/2020/12/23/ccls/  Nvim, I think
https://stackoverflow.com/questions/62461375/clang-using-cmake-to-build-a-compile-commands-json-for-my-project
https://www.reddit.com/r/cmake/comments/u0ztoq/compile_commandsjson_from_cmakelist/
https://clang.llvm.org/docs/HowToSetupToolingForLLVM.html

Then there is this youtube video I happened to catch recently.

https://www.youtube.com/watch?v=nRpvWgymOuc&t=2467s  7:30

But I guess you can make major modes (or user code) or minor-modes
set eglot-server-programs or add methods to
eglot-initialization-options in clever ways to automatically
point clangd to a given compile-commands.json.  And quite likely
you can convince CMake to create it in other places too.  But
I've never needed to, the 'ln' solution is too good.  You can even
commit the symlink.

> What I'm wondering is whether we might be currently disadvantaged
> compared to the users of some other editors, which might have some more
> auto-detection in that area.

Dunno.  At $DAYJOB there are people struggling in VSCode extension
mayhem with faulty static analysis because they have no idea
that the compile_commands.json is a config-time artifact, maybe
the plugin tries to hide that from them?

But I wouldn't be surprised if lsp-mode features this in its
c++-specific file.  Haven't checked.  My policy has always been to keep
language-specific cruft like this out of Eglot, make the common case
easy and allow the fancy case to be coded in .emacs, in major-mode or
libraries.

> I'm open to continuing this discussion somewhere

Don't bother :-) I've given all my feedback on that matter, don't
have time, am happy with my .emacs hacks and so are those Eglot users.