bug#65518: 30.0.50; [FR Eglot] Completions over all workspace symbols

bug#65518: 30.0.50; [FR Eglot] Completions over all workspace symbols

[Gerd Möllmann]

I'm in a C buffer with Eglot running, and I want to go to the definition
of something whose name I know, but point is not on an identifier naming
that thing.  For instance, I want to go the definition of Lisp_Package
(in an obsure branch having a struct Lisp_Package), while point is
somewhere.

C-u M-. offers some completions, but Lisp_Package is not among them.
Not using a completion and entering Lisp_Package works.

Feature request: Could the completions be made to contain all workspace
symbols. maybe as a n optional feature?  Lsp-mode has something doing
that, and I found it quite useful with Helm.

In GNU Emacs 30.0.50 (build 2, aarch64-apple-darwin22.6.0, NS
 appkit-2299.70 Version 13.5 (Build 22G74)) of 2023-08-24 built on
 Mini.fritz.box
Repository revision: 53c07bd04bf59f63e49af2c626714bf3fdd03ad6
Repository branch: scratch/pkg
Windowing system distributor 'Apple', version 10.3.2299
System Description:  macOS 13.5

bug#65518: 30.0.50; [FR Eglot] Completions over all workspace symbols

[João Távora]

Gerd Möllmann <gerd.moellmann@gmail.com> writes:

> I'm in a C buffer with Eglot running, and I want to go to the definition
> of something whose name I know, but point is not on an identifier naming
> that thing.  For instance, I want to go the definition of Lisp_Package
> (in an obsure branch having a struct Lisp_Package), while point is
> somewhere.

>
> C-u M-. offers some completions, but Lisp_Package is not among them.
> Not using a completion and entering Lisp_Package works.

Like I explained in bug#65517 (which see), this too is based on the
'workspace/symbol' request.  You can check your Eglot events buffer (M-x
eglot-events-buffer) around the time you press C-u M-. to get a feel of
how the server responds to these queries.

You'll see Eglot sends the widest possible 'query' to the LSP server
which returns a limited number of names (clangd caps at exactly 100 by
default).  In large projects it's unlikely the symbol you're looking for
is there, but as you type more text into the prompt, further
'workspace/symbol' requests are made.

Note that, similarly to many other LSP situations, Eglot and Emacs has
no control over _how_ the server interprets the 'query' string, so there
is no control over the "completion style" and your setting of
'completion-style' has no effect here. The clangd LSP server in
particular seems to use a somewhat "flex" or "fuzzy" completion style.

You didn't specify exactly how you entered input into your search box,
and what completion package, if any, you are using.  So, whereas in
bug#65517 it was simple to understand the root cause of the observed
behavior, in this particular case, I'm afraid I'm going to have to refer
you to the usual checklist on making reproducible bug reports for Eglot:

  https://joaotavora.github.io/eglot/#Troubleshooting-Eglot

Here, for your specific example, I'm guessing the project is Emacs
itself (checked out in your CL packages branch).  I'm also guessing
you're using the `clangd` server and some of compile_commands.json.

Confirming this data and specifying versions etc should be enough for me
to reproduce the problem exactly as it appeared to you.  I'm also
guessing (or at least hoping), you don't use complicated third party
completion packages and everything can be experimented with via Emacs
-Q.

For reference, C-u M-. works fairly well here in my projects when used
with clangd 15.0.7 and the compile_commands.json file generated by the
CMake system.  I don't yet fully understand how clangd's sorting
heuristics and completion pattern matching work, but I seem to
eventually reach the name I want.

João

bug#65518: 30.0.50; [FR Eglot] Completions over all workspace symbols

[Gerd Möllmann]

João Távora <joaotavora@gmail.com> writes:

> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>
>> I'm in a C buffer with Eglot running, and I want to go to the definition
>> of something whose name I know, but point is not on an identifier naming
>> that thing.  For instance, I want to go the definition of Lisp_Package
>> (in an obsure branch having a struct Lisp_Package), while point is
>> somewhere.
>
>>
>> C-u M-. offers some completions, but Lisp_Package is not among them.
>> Not using a completion and entering Lisp_Package works.
>
> Like I explained in bug#65517 (which see), this too is based on the
> 'workspace/symbol' request.  You can check your Eglot events buffer (M-x
> eglot-events-buffer) around the time you press C-u M-. to get a feel of
> how the server responds to these queries.
>
> You'll see Eglot sends the widest possible 'query' to the LSP server
> which returns a limited number of names (clangd caps at exactly 100 by
> default).  In large projects it's unlikely the symbol you're looking for
> is there, but as you type more text into the prompt, further
> 'workspace/symbol' requests are made.
>
> Note that, similarly to many other LSP situations, Eglot and Emacs has
> no control over _how_ the server interprets the 'query' string, so there
> is no control over the "completion style" and your setting of
> 'completion-style' has no effect here. The clangd LSP server in
> particular seems to use a somewhat "flex" or "fuzzy" completion style.
>
> You didn't specify exactly how you entered input into your search box,
> and what completion package, if any, you are using.  So, whereas in
> bug#65517 it was simple to understand the root cause of the observed
> behavior, in this particular case, I'm afraid I'm going to have to refer
> you to the usual checklist on making reproducible bug reports for Eglot:
>
>   https://joaotavora.github.io/eglot/#Troubleshooting-Eglot
>
> Here, for your specific example, I'm guessing the project is Emacs
> itself (checked out in your CL packages branch).  I'm also guessing
> you're using the `clangd` server and some of compile_commands.json.
>
> Confirming this data and specifying versions etc should be enough for me
> to reproduce the problem exactly as it appeared to you.  I'm also
> guessing (or at least hoping), you don't use complicated third party
> completion packages and everything can be experimented with via Emacs
> -Q.
>
> For reference, C-u M-. works fairly well here in my projects when used
> with clangd 15.0.7 and the compile_commands.json file generated by the
> CMake system.  I don't yet fully understand how clangd's sorting
> heuristics and completion pattern matching work, but I seem to
> eventually reach the name I want.

We are miscommunicating.  This is not a bug report, it's a feature
request. Referring me to a description of how to produce reproducible
bug reports I therefore find odd.

As I mentioned, lsp-mode offers the functionality, that I'd like to see in
Eglot.  I don't know how it does it, and I don't care, really.

bug#65518: 30.0.50; [FR Eglot] Completions over all workspace symbols

[João Távora]

On Sat, Aug 26, 2023 at 6:09 AM Gerd Möllmann <gerd.moellmann@gmail.com> wrote:
>
> João Távora <joaotavora@gmail.com> writes:
>
> > Gerd Möllmann <gerd.moellmann@gmail.com> writes:
> >

> We are miscommunicating.  This is not a bug report, it's a feature
> request. Referring me to a description of how to produce reproducible
> bug reports I therefore find odd.


Please don't find it odd.  And we are miscommunicating indeed.

Let me be clear: as far as I can detect, and as far as I can understand
it, the situation you describe shouldn't exist.  The pattern `LiPa` should
eventually allow the completion `Lisp_Package` to be offered in theory.

I reach all my completions with similar abbreviated patterns for
example.

But perhaps you aren't using Eglot how I think you are using it.
Perhaps you are using some older clangd version (known to miss completions)
or perhaps you are using some popular "Corfu" package (also known
to miss completions).  Or perhaps there is a indeed bug in Eglot.

Therefore, to give you an and others readers a useful answer,
I must have more information.  And providing the information
in that recipe is the standard way I ask of all Eglot users.

So to continue the analysis of this, I beg you to provide it.
If you don't want to go through that list, I've already collected
some information, so it boils down to telling me:

1. Emacs version and Eglot version
2. Clangd version
3. How to produce your compile_commands.json file
4. What branch of the Emacs C source this is
5. A short description of exactly what you do after starting
Emacs -Q, what you see, and what you would like to see instead.

> As I mentioned, lsp-mode offers the functionality, that I'd like to see in
> Eglot.  I don't know how it does it, and I don't care, really.

If you want a comparative analysis to lsp-mode, to increase
the chances that this is addressed, I'd also need to know exactly
how you set it up.  As far as I understand, lsp-mode doesn't
do miracles, it uses the same LSP interfaces available to Eglot
(in this case, more than likely `symbol/workspace`, which I've
already described).

João

bug#65518: 30.0.50; [FR Eglot] Completions over all workspace symbols

[João Távora]

João Távora <joaotavora@gmail.com> writes:

> If you want a comparative analysis to lsp-mode, to increase
> the chances that this is addressed, I'd also need to know exactly
> how you set it up.  As far as I understand, lsp-mode doesn't
> do miracles, it uses the same LSP interfaces available to Eglot
> (in this case, more than likely `symbol/workspace`, which I've
> already described).

Out of curiosity, I've had a look at lsp-mode.el.  It does not use
'symbol/workspace', rather 'textDocument/documentSymbol'.  So C-u
M-. provides a complete list of symbols upfront.  But -- crucially --
that list is limited to contain symbols for the current document only .

So, at first glance, I'd say it's lsp-mode who is missing (lots and lots
of) symbols.

In contrast, Eglot's C-u M-. reaches all the symbols in the whole
project, a much better implementation of 'xref-find-deinitions' IMNSHO.
In a large project, like LLVM, using Eglot in combination with something
like `M-x fido-vertical-mode` gives you an interactive incremental
completion interface that is able reach millions of symbols across in
thousands of C++ files.

João

bug#65518: 30.0.50; [FR Eglot] Completions over all workspace symbols

[Gerd Möllmann]

João Távora <joaotavora@gmail.com> writes:

> João Távora <joaotavora@gmail.com> writes:
>
>> If you want a comparative analysis to lsp-mode, to increase
>> the chances that this is addressed, I'd also need to know exactly
>> how you set it up.  As far as I understand, lsp-mode doesn't
>> do miracles, it uses the same LSP interfaces available to Eglot
>> (in this case, more than likely `symbol/workspace`, which I've
>> already described).
>
> Out of curiosity, I've had a look at lsp-mode.el.  It does not use
> 'symbol/workspace', rather 'textDocument/documentSymbol'.  So C-u
> M-. provides a complete list of symbols upfront.  But -- crucially --
> that list is limited to contain symbols for the current document only .
>
> So, at first glance, I'd say it's lsp-mode who is missing (lots and lots
> of) symbols.
>
> In contrast, Eglot's C-u M-. reaches all the symbols in the whole
> project, a much better implementation of 'xref-find-deinitions' IMNSHO.
> In a large project, like LLVM, using Eglot in combination with something
> like `M-x fido-vertical-mode` gives you an interactive incremental
> completion interface that is able reach millions of symbols across in
> thousands of C++ files.

Interesting.  Maybe I never noticed this because Emacs' C files include
so much that I got the impression that that was everything.

Thanks for testing this!

I guess we can close this request, then.  I'll see if can use etags, or
something, for that use-case.  Or just live with it.

Thanks!

bug#65518: 30.0.50; [FR Eglot] Completions over all workspace symbols

[João Távora]

On Sun, Aug 27, 2023 at 6:56 AM Gerd Möllmann <gerd.moellmann@gmail.com> wrote:

> > In contrast, Eglot's C-u M-. reaches all the symbols in the whole
> > project, a much better implementation of 'xref-find-deinitions' IMNSHO.
> > In a large project, like LLVM, using Eglot in combination with something
> > like `M-x fido-vertical-mode` gives you an interactive incremental
> > completion interface that is able reach millions of symbols across in
> > thousands of C++ files.
>
> Interesting.  Maybe I never noticed this because Emacs' C files include
> so much that I got the impression that that was everything.
>
> Thanks for testing this!
>
> I guess we can close this request, then.  I'll see if can use etags, or
> something, for that use-case.  Or just live with it.
>
> Thanks!

I'm glad to help, but I must say I'm still at a loss of what the use-case
actually is.  As far as I can understand, at least from the subject of
this email, "completion over all workspace symbols" is very much a
supported feature in Eglot.  What exactly are you missing?  May I
ask what completion UI package you are using?  Is it Helm? Vertico?
Corfu? Ivy? Or something else?

I primarily test and use Eglot with Emacs itself.  I use the built-in
completion package fido-vertical-mode.  The only relevant third-party package
I use is company-mode, but it's not directly relevant here.

João

bug#65518: 30.0.50; [FR Eglot] Completions over all workspace symbols

[Gerd Möllmann]

João Távora <joaotavora@gmail.com> writes:

> I'm glad to help, but I must say I'm still at a loss of what the use-case
> actually is.  As far as I can understand, at least from the subject of
> this email, "completion over all workspace symbols" is very much a
> supported feature in Eglot.  What exactly are you missing?  May I
> ask what completion UI package you are using?  Is it Helm? Vertico?
> Corfu? Ivy? Or something else?
>
> I primarily test and use Eglot with Emacs itself.  

Me too.

> I use the built-in completion package fido-vertical-mode.  The only
> relevant third-party package I use is company-mode, but it's not
> directly relevant here.

I'm currently using Company + Helm.  But it's changing every now and then.

bug#65518: 30.0.50; [FR Eglot] Completions over all workspace symbols

[Dmitry Gutov]

On 25/08/2023 12:22, João Távora wrote:
> You'll see Eglot sends the widest possible 'query' to the LSP server
> which returns a limited number of names (clangd caps at exactly 100 by
> default).

Perhaps it's worth documenting how this value could be increased 
(argument --limit-results=, apparently).

Unfortunately, it's currently shared with code completion (proposal to 
change that here: https://github.com/clangd/clangd/discussions/761), so 
raising it too much could made code completion jerkier.

bug#65518: 30.0.50; [FR Eglot] Completions over all workspace symbols

[Gerd Möllmann]

On 28.08.23 02:19, Dmitry Gutov wrote:
> On 25/08/2023 12:22, João Távora wrote:
>> You'll see Eglot sends the widest possible 'query' to the LSP server
>> which returns a limited number of names (clangd caps at exactly 100 by
>> default).
> 
> Perhaps it's worth documenting how this value could be increased 
> (argument --limit-results=, apparently).

I've tried that now, and it works pretty nicely for me.  Apple clang 
version 14.0.3 (clang-1403.0.22.14.1)

I now have

export CLANGD_FLAGS="--limit-results=1000 --limit-references=1000"

in my .zshenv, and I'm using exec-path-from-shell in Emacs to get
that setting into Emacs on macOS.

Also, and completely unrelatad, I found that one can add a .clangd, 
which made post-processing the compile_commands.json generated by bear 
unneccessary.

Put a .clangd in the toplevel directory of the Emacs tree with contents

---
If:
     PathMatch: "src/.*\.c"
CompileFlags:
     Add: [-include=config.h]

(Not sure if that first line is obligatory.  I'm not a YAML user.)