bug#72088: eglot: Segregates language servers in a way that hampers polyglot language servers

bug#72088: eglot: Segregates language servers in a way that hampers polyglot language servers

[Niels Thykier]

Hi

Thanks for providing LSP support in emacs. :)


For reference, I used emacs 29.4 from Debian when I tested. If this 
feature has changed in emacs 30, then I would not have seen it yet (I am 
not tracking emacs/eglot development a lot).


I am working on a language server for Debian packaging (called 
`debputy`). The Debian packaging files consists of several different 
file formats with cross references between the formats. As a 
consequence, I built my language server to be a "polyglot" (multi-language).

While testing with eglot, I noticed that my language server was not 
informed of all changes. As far as I can see, eglot spawns a language 
server per file format (major mode) and then only informs the language 
server of events (didOpen/didChange) for that major mode. This happens 
even through it is exactly the same command + args for both major modes.

The next result is that my language server when used via emacs + eglot, 
requires the user to save a file before the language server can "see" 
the effect when you are working in a different file.

I was a bit in doubt whether the spec accounted for polyglot language 
servers, so I asked in the LSP github for clarification. They made the 
argument that a language server should be able to see events for other 
file formats and suggested that I contacted you (the eglot maintainers) 
about this 
(https://github.com/microsoft/language-server-protocol/issues/1964). 
They acknowledged this was implicit and then listed the two key features 
they saw as supporting this implicit requirement (both of them boils 
down to DocumentSelector as I read it).

I tried a static registration with a DocumentSelector on `**/debian/*` 
(I think it was hover docs, though it might have been completion), but 
my debugging suggested that my instances did not get didOpen/didChange 
events "across" the file formats even with this registration.

Can we work towards a solution where I can ask eglot to provide 
didOpen/didChange events across formats? :)

Best regards,
Niels

bug#72088: eglot: Segregates language servers in a way that hampers polyglot language servers

[Eli Zaretskii]

> Date: Fri, 12 Jul 2024 13:33:56 +0200
> From: Niels Thykier <niels@thykier.net>
> 
> Hi
> 
> Thanks for providing LSP support in emacs. :)
> 
> 
> For reference, I used emacs 29.4 from Debian when I tested. If this 
> feature has changed in emacs 30, then I would not have seen it yet (I am 
> not tracking emacs/eglot development a lot).
> 
> 
> I am working on a language server for Debian packaging (called 
> `debputy`). The Debian packaging files consists of several different 
> file formats with cross references between the formats. As a 
> consequence, I built my language server to be a "polyglot" (multi-language).
> 
> While testing with eglot, I noticed that my language server was not 
> informed of all changes. As far as I can see, eglot spawns a language 
> server per file format (major mode) and then only informs the language 
> server of events (didOpen/didChange) for that major mode. This happens 
> even through it is exactly the same command + args for both major modes.
> 
> The next result is that my language server when used via emacs + eglot, 
> requires the user to save a file before the language server can "see" 
> the effect when you are working in a different file.
> 
> I was a bit in doubt whether the spec accounted for polyglot language 
> servers, so I asked in the LSP github for clarification. They made the 
> argument that a language server should be able to see events for other 
> file formats and suggested that I contacted you (the eglot maintainers) 
> about this 
> (https://github.com/microsoft/language-server-protocol/issues/1964). 
> They acknowledged this was implicit and then listed the two key features 
> they saw as supporting this implicit requirement (both of them boils 
> down to DocumentSelector as I read it).
> 
> I tried a static registration with a DocumentSelector on `**/debian/*` 
> (I think it was hover docs, though it might have been completion), but 
> my debugging suggested that my instances did not get didOpen/didChange 
> events "across" the file formats even with this registration.
> 
> Can we work towards a solution where I can ask eglot to provide 
> didOpen/didChange events across formats? :)

João, any comments or suggestions?

bug#72088: eglot: Segregates language servers in a way that hampers polyglot language servers

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 2601 bytes --]

Major modes in eglot-server-programs don't work? What was attempted? I
couldn't understand. Look at existing examples, like C/C++ file types being
handled by the same server, and others.

João

On Sat, Jul 13, 2024, 07:16 Eli Zaretskii <eliz@gnu.org> wrote:

> > Date: Fri, 12 Jul 2024 13:33:56 +0200
> > From: Niels Thykier <niels@thykier.net>
> >
> > Hi
> >
> > Thanks for providing LSP support in emacs. :)
> >
> >
> > For reference, I used emacs 29.4 from Debian when I tested. If this
> > feature has changed in emacs 30, then I would not have seen it yet (I am
> > not tracking emacs/eglot development a lot).
> >
> >
> > I am working on a language server for Debian packaging (called
> > `debputy`). The Debian packaging files consists of several different
> > file formats with cross references between the formats. As a
> > consequence, I built my language server to be a "polyglot"
> (multi-language).
> >
> > While testing with eglot, I noticed that my language server was not
> > informed of all changes. As far as I can see, eglot spawns a language
> > server per file format (major mode) and then only informs the language
> > server of events (didOpen/didChange) for that major mode. This happens
> > even through it is exactly the same command + args for both major modes.
> >
> > The next result is that my language server when used via emacs + eglot,
> > requires the user to save a file before the language server can "see"
> > the effect when you are working in a different file.
> >
> > I was a bit in doubt whether the spec accounted for polyglot language
> > servers, so I asked in the LSP github for clarification. They made the
> > argument that a language server should be able to see events for other
> > file formats and suggested that I contacted you (the eglot maintainers)
> > about this
> > (https://github.com/microsoft/language-server-protocol/issues/1964).
> > They acknowledged this was implicit and then listed the two key features
> > they saw as supporting this implicit requirement (both of them boils
> > down to DocumentSelector as I read it).
> >
> > I tried a static registration with a DocumentSelector on `**/debian/*`
> > (I think it was hover docs, though it might have been completion), but
> > my debugging suggested that my instances did not get didOpen/didChange
> > events "across" the file formats even with this registration.
> >
> > Can we work towards a solution where I can ask eglot to provide
> > didOpen/didChange events across formats? :)
>
> João, any comments or suggestions?
>

[-- Attachment #2: Type: text/html, Size: 3410 bytes --]

bug#72088: eglot: Segregates language servers in a way that hampers polyglot language servers

[Eli Zaretskii]

> From: João Távora <joaotavora@gmail.com>
> Date: Sat, 13 Jul 2024 17:29:04 +0100
> Cc: Niels Thykier <niels@thykier.net>, 72088@debbugs.gnu.org
> 
> Major modes in eglot-server-programs don't work? What was attempted? I couldn't understand. Look at
> existing examples, like C/C++ file types being handled by the same server, and others.

AFAIU, the OP tried to use the same LSP server for several languages,
or something like that.

bug#72088: eglot: Segregates language servers in a way that hampers polyglot language servers

[Niels Thykier]

Eli Zaretskii:
>> From: João Távora <joaotavora@gmail.com>
>> Date: Sat, 13 Jul 2024 17:29:04 +0100
>> Cc: Niels Thykier <niels@thykier.net>, 72088@debbugs.gnu.org
>>
>> Major modes in eglot-server-programs don't work? What was attempted? I couldn't understand. Look at
>> existing examples, like C/C++ file types being handled by the same server, and others.
> 
> AFAIU, the OP tried to use the same LSP server for several languages,
> or something like that.

Correct, I am using the same LSP server for several languages (major-modes).

I observe that eglot spawns an instance *per* major mode even though I 
use exactly the same definition in eglot-server-programs. Additionally, 
each instance only receives "didOpen" and "didChange" events for the 
given mode they were spawned for.

 From my PoV, ideally, it would be possible to ask eglot to spawn only 
one instance across all major modes and it would receive "didOpen" and 
"didChange" events for all the modes that it was registered for. Like 
the singleton pattern from programming languages.


Alternatively, I think it would also be possible to have an instance per 
major mode as long as they receive the "didOpen" and "didChange" for all 
relevant modes (that is, eglot sends the same "didOpen"/"didChange" 
event to multiple servers). The downside of the later approach is that 
the same computations happen twice and is sent back to eglot that then 
has deal with the duplication. Notably push diagnostics would then 
happen once per server per major mode.


Note since I am the author of the language server, I am looking for a 
way for eglot manage the life-cycle of the language server. Asking my 
consumers to manually start and stop the language server would make it a 
second-class language server at best.

Best regards,
Niels

bug#72088: eglot: Segregates language servers in a way that hampers polyglot language servers

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 685 bytes --]

On Sun, Jul 14, 2024, 05:37 Eli Zaretskii <eliz@gnu.org> wrote:

> AFAIU, the OP tried to use the same LSP server for several languages,
> or something like that.
>

Yes, but how? What are the major modes in question? What information was
added to eglot-server-programs, if any? Are the two different files in
question in the same project?The OP consistently conflates file type and
major mode, which is not always true. A simple example of a LSP server that
manages different files in (slightly) different major modes (as long as in
the same project) is clangd. But there are a few others, I think. The
docstring of eglot-server-programs should be studied.

João

>

[-- Attachment #2: Type: text/html, Size: 1230 bytes --]

bug#72088: eglot: Segregates language servers in a way that hampers polyglot language servers

[Niels Thykier]

João Távora:
> On Sun, Jul 14, 2024, 05:37 Eli Zaretskii <eliz@gnu.org> wrote:
> 
>> AFAIU, the OP tried to use the same LSP server for several languages,
>> or something like that.
>>
> 
> Yes, but how? What are the major modes in question? What information was
> added to eglot-server-programs, if any?
Hi,

The `eglot-server-programs` are defined as:

```
(with-eval-after-load 'eglot
     (add-to-list 'eglot-server-programs
             '(debian-control-mode . ("debputy" "lsp" "server")))
     (add-to-list 'eglot-server-programs
             '(debian-changelog-mode . ("debputy" "lsp" "server")))
     (add-to-list 'eglot-server-programs
             '(debian-copyright-mode . ("debputy" "lsp" "server")))
     (add-to-list 'eglot-server-programs
             '(makefile-gmake-mode . ("debputy" "lsp" "server")))
     (add-to-list 'eglot-server-programs
             '(yaml-mode . ("debputy" "lsp" "server")))
     (add-to-list 'eglot-server-programs
             '(debian-autopkgtest-control-mode . ("debputy" "lsp" 
"server")))
)
```

Concrete case I was debugging was a `debian/control` file 
(`debian-control-mode`) + a `debian/rules` file (`makefile-qmake-mode`).

The "debian-X" modes is coming from 
https://salsa.debian.org/emacsen-team/dpkg-dev-el. I did not remember 
actively installing the `makefile-qmake-mode`, so I assume it is bundled 
with emacs. The "yaml-mode" is elpa-yaml-mode 
(https://github.com/yoshiki/yaml-mode).

Note: The `makefile-gmake-mode` and `yaml-mode` registrations would 
ideally be restricted to specific file names (such as `debian/rules` and 
`debian/debputy.manifest`). Mentioning it mostly for the context (I have 
not looked into how to do that at an eglot level yet).

> Are the two different files in question in the same project?

Yes, as far as I can tell, eglot considers them to be in the same 
project. Concretely, the files are both in a .git repo and eglot reports 
the git root dir as a workspace folder to the server.

> The OP consistently conflates file type and
> major mode, which is not always true.

Apologies for that. The language server in question works with at least 
4 distinct syntactical formats (YAML, make, the debian changelog format 
plus 3 distinct deb822 files). You are correct that the 3 deb822 files 
do have different major modes despite being the same file format at a 
syntactical level.

Nevertheless, in the concrete case, the problem appears between two 
distinct file formats *and* major-modes.

> A simple example of a LSP server that
> manages different files in (slightly) different major modes (as long as in
> the same project) is clangd. But there are a few others, I think. The
> docstring of eglot-server-programs should be studied.
> 
> João
> 
>>
> 

I read through 
https://www.gnu.org/software/emacs/manual/html_node/eglot/Setting-Up-LSP-Servers.html 
and I do not see anything here that decides how to control whether one 
or two instances per project will be spawned.

Best regards,
Niels

bug#72088: eglot: Segregates language servers in a way that hampers polyglot language servers

[Niels Thykier]

Niels Thykier:
> João Távora:
>> On Sun, Jul 14, 2024, 05:37 Eli Zaretskii <eliz@gnu.org> wrote:
>>
>>> AFAIU, the OP tried to use the same LSP server for several languages,
>>> or something like that.
>>>
>>
>> Yes, but how? What are the major modes in question? What information was
>> added to eglot-server-programs, if any?
> Hi,
> 
> The `eglot-server-programs` are defined as:
> 
> ```
> (with-eval-after-load 'eglot
>      (add-to-list 'eglot-server-programs
>              '(debian-control-mode . ("debputy" "lsp" "server")))
>      (add-to-list 'eglot-server-programs
>              '(debian-changelog-mode . ("debputy" "lsp" "server")))
>      (add-to-list 'eglot-server-programs
>              '(debian-copyright-mode . ("debputy" "lsp" "server")))
>      (add-to-list 'eglot-server-programs
>              '(makefile-gmake-mode . ("debputy" "lsp" "server")))
>      (add-to-list 'eglot-server-programs
>              '(yaml-mode . ("debputy" "lsp" "server")))
>      (add-to-list 'eglot-server-programs
>              '(debian-autopkgtest-control-mode . ("debputy" "lsp" 
> "server")))
> )
> ```
> 
> [...]
> 
> I read through 
> https://www.gnu.org/software/emacs/manual/html_node/eglot/Setting-Up-LSP-Servers.html and I do not see anything here that decides how to control whether one or two instances per project will be spawned.
> 
> Best regards,
> Niels
> 

No, I think I see it now.

I think I have overlooked this paragraph:

"""
The major-mode of the alist elements can be either a symbol of an Emacs 
major mode or a list of the form (mode :language-id id), with mode being 
a major-mode symbol and id a string that identifies the language to the 
server (if Eglot cannot by itself convert the major-mode to the language 
identifier string required by the server). In addition, major-mode can 
be a list of several major modes specified in one of the above forms – 
this means a running instance of the associated server is responsible 
for files of multiple major modes or languages in the project.
"""

I will retry with this in mind and see if that solves the problem. If it 
does, I will close this bug.

Best regards,
Niels

bug#72088: eglot: Segregates language servers in a way that hampers polyglot language servers

[Niels Thykier]

Niels Thykier:
> [...]
> 
> No, I think I see it now.
> 
> I think I have overlooked this paragraph:
> 
> """
> The major-mode of the alist elements can be either a symbol of an Emacs 
> major mode or a list of the form (mode :language-id id), with mode being 
> a major-mode symbol and id a string that identifies the language to the 
> server (if Eglot cannot by itself convert the major-mode to the language 
> identifier string required by the server). In addition, major-mode can 
> be a list of several major modes specified in one of the above forms – 
> this means a running instance of the associated server is responsible 
> for files of multiple major modes or languages in the project.
> """
> 
> I will retry with this in mind and see if that solves the problem. If it 
> does, I will close this bug.
> 
> Best regards,
> Niels
> 

Indeed, the list form of `major-mode` works, closing.

Thanks for the hint about `clang`, which gave me the concrete syntax I 
needed.

Best regards,
Niels

bug#72088: eglot: Segregates language servers in a way that hampers polyglot language servers

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 876 bytes --]

On Sun, Jul 14, 2024, 10:36 Niels Thykier <niels@thykier.net> wrote:

> I read through
>
> https://www.gnu.org/software/emacs/manual/html_node/eglot/Setting-Up-LSP-Servers.html
> and I do not see anything here that decides how to control whether one
> or two instances per project will be spawned.
>

In that section, did you happen to read this sentence?

"In addition, major-mode can be a list of several major modes specified in
one of the above forms – this means a running instance of the associated
server is responsible for files of multiple major modes or languages in the
project. "

If so, how do you comprehend or interpret it? Maybe it could be clarified.
Regardless, the default value of the eglot-server-programs variable that
ships with Emacs has several examples that make use of this feature, so it
may be worth studying those.

João

[-- Attachment #2: Type: text/html, Size: 1532 bytes --]

bug#72088: eglot: Segregates language servers in a way that hampers polyglot language servers

[Niels Thykier]

João Távora:
> On Sun, Jul 14, 2024, 10:36 Niels Thykier <niels@thykier.net> wrote:
> 
>> I read through
>>
>> https://www.gnu.org/software/emacs/manual/html_node/eglot/Setting-Up-LSP-Servers.html
>> and I do not see anything here that decides how to control whether one
>> or two instances per project will be spawned.
>>
> 
> In that section, did you happen to read this sentence?
> 
> "In addition, major-mode can be a list of several major modes specified in
> one of the above forms – this means a running instance of the associated
> server is responsible for files of multiple major modes or languages in the
> project. "
> 
> If so, how do you comprehend or interpret it? Maybe it could be clarified.
> Regardless, the default value of the eglot-server-programs variable that
> ships with Emacs has several examples that make use of this feature, so it
> may be worth studying those.
> 
> João
> 

I missed it entirely.

My recommendations would one or more of the following:

  1 Use a more verbose variant of the `major-mode` placeholder, like
    `(major-mode-or-modes . server)`. This would have prompted me
    to not assume `major-mode` was just a single major-mode.

  2 Provide an example featuring multiple modes in addition the existing
    examples. This example could also show off the `:language-id`
    feature.

  3 Provide a howto guide for adding a new LSP server configuration
    that explicitly featured the distinction between single mode vs.
    multi-mode LSP servers.
    (Note "howto guide" is specifically the "howto guide" from
     https://docs.divio.com/documentation-system/)

For me, either 1+2 would definitely have worked. The third option is a 
question of which kind of audience you want to optimize for. Personally, 
I find it difficult myself to figure out what deserves a howto guide, so 
I can totally understand if you pass on doing one of those.



A bit more on what happened on my end:


I think a key aspect of this problem is that I was trying to quickly 
solve a problem. Namely getting my prototype to work and all I could 
find was reference documentation, which is tailored for solving a 
different type of documentation problem than what I ideally needed 
(Compare problem-oriented vs information-oriented on 
https://docs.divio.com/documentation-system/).
   Since I know the documentation has the "wrong focus" for the thing I 
want to do, I scan much more aggressively for examples and highlighted 
words that looks like definitions and then retrace from there if 
relevant. This is a common issue for many projects including some of my 
own that they only provide one or two types of documentation (with 
reference documentation being the most common type of documentation), 
which has made me adopt this reading style.


The use of `major-mode` in the documentation had me assume that it was 
one major mode at the time which works out of the box with no warnings. 
The moment I saw: "The value of the variable is an alist, whose elements 
are of the form (major-mode . server)", I was like "Got it, one major 
mode + a server definition. Where do I find the definition of `server`? 
Next line talks about the major-mode and I got what is. I need the 
server part, oh I see the highlighted `(programs args)` below, so skip 
to that. Oh, there are examples in the bottom. Ok, so 5x the first 
example that with a bit of replacements" and I had some code that worked 
and I was back to working on my language server. From there it took me 
several months until the server reached maturity enough to provide 
features where this mistake got in the way.


As mentioned, an example that might have worked would have been:

```
;; Polyglot LSP server example. The instance is reused
;; in a given project for all the listed modes.
(with-eval-after-load 'eglot
   (add-to-list 'eglot-server-programs
                '((
                   c-mode
                   (cpp-mode :language-id "c++"))
                   . ("custom-c-lsp" "--stdio"))))
```


In fact, it would have been nice to have this example, because I spent 
30 minutes fighting with the elisp syntax figuring out the syntax. After 
that, I spent 15 minutes trying to figure out why my `eglot-ensure` 
configuration stopped working (as you can probably tell, lisp/elisp is 
not really not my jam).

Hope that helped.

Best regards,
Niels

bug#72088: eglot: Segregates language servers in a way that hampers polyglot language servers

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 516 bytes --]

On Sun, Jul 14, 2024, 13:01 Niels Thykier <niels@thykier.net> wrote:

This is too long for me to address, but Eglot's documentation (and all of
Eglot) follows the simple principle of making the simple common thing easy
and the complex uncommon thing possible. Anyway, when you spent a little
bit more time reading, you found the paragraph even before I pointed it out
or to you, which I count as very good. If you want to submit a patch with
that example, it's welcome so long as it integrates into the current doc.

[-- Attachment #2: Type: text/html, Size: 764 bytes --]