Xref/tags/lsp possible bug

Xref/tags/lsp possible bug

[Alessandro Bertulli]

Hi all!

I just stumbled upon a very strange behavior of Emacs, I don't know if
it is a bug or an error of my configuration. Basically, I noticed two
things:

- First, when using LSP modes (both lsp-mode and eglot), the command
  xref-find-definitions only acts on the symbol under point. I
  explain: usually, in vanilla Emacs (emacs -q) if point is on a
  whitespace/not on a symbol, invoking the command (bound to M-.) will
  prompt you for an identifier to search for. However, when LSP is
  active, it directly searches for an empty symbol, for example
  reporting something similar to: "Symbol  not found" (not the double
  space, which makes me think it searches for a symbol named ""). As I
  said, this happend both for lsp-mode and eglot, so maybe is a
problem of xref?

- Second, and minor: while I was investigating, I noticed that,
  depending on the loaded packages, when giving M-., xref sometimes
  asks for the symbol to search, sometimes it asks the TAGS table
first. What package can be the problem?

Do you think it is a bug of Emacs? Do you know a way to resolve?
Thanks to all!

############
I report some useful infos:

Emacs version: This is GNU Emacs 28.1 (build 1, x86_64-pc-linux-gnu, GTK+
Version 3.24.33, cairo version 1.17.6) of 2022-04-20
lsp-mode version: 20220422.2059
xref version: 1.4.1

##########
Steps to reproduce:
-First bug:
 - launch emacs -q
 - go on an C++ file
 - on a whitespace, give M-.. The prompt says "Find definition of: "
 - close and restart with the provided file (emacs -q --load file,
  see end of the mail)
 - go on the same C++ file
 - launch lsp-mode (M-x lsp)
 - on the same whitespace, give M-. The minibuffer says "Unable to
 find symbol  in current document" (note the bouble space). In
 *Messages* buffer, the error is reported perfized with the string
 "user-error: ".
 - give M-x lsp-find-definition on the same whitespace. The minibuffer
 says: "LSP :: Not found for: " (note the space at the end).


-Second bug:
 - launch emacs -q
 - go on a C++ file
 - on a whitespace, give M-. The prompt says "Find definitions of: "
 - close and restart with provided file (emacs -q --load file)
 - go on the same C++ file
 - on the same whitespace, give M-. The prompt says "Visit tags table
   (default TAGS): "
 - go on an elisp file, do the same, the prompt says "Find definitions of: "

#############
Test init file

(custom-set-variables
 ;; custom-set-variables was added by Custom.
 ;; If you edit it by hand, you could mess it up, so be careful.
 ;; Your init file should contain only one such instance.
 ;; If there is more than one, they won't work right.
 '(package-selected-packages
   '(helpful transient-dwim ivy projectile eglot magit lsp-ui flycheck
which-key yasnippet-snippets yasnippet lsp-mode company))
 )

;;--------------
;; Enable MELPA
;;-------------

(require 'package)
(add-to-list 'package-archives '("melpa" . "https://melpa.org/packages/") t)
;; Comment/uncomment this line to enable MELPA Stable if desired.  See
`package-archive-priorities`
;; and `package-pinned-packages`. Most users will not need or want to do
this.
;;(add-to-list 'package-archives '("melpa-stable" . "
https://stable.melpa.org/packages/") t)
(package-initialize)


(setq pkg-list '(helpful transient-dwim ivy projectile eglot magit lsp-ui
flycheck which-key yasnippet-snippets yasnippet lsp-mode company))
(mapc (lambda (pkg)
          (unless (package-installed-p pkg)
            (package-install pkg))
          (require pkg))
      pkg-list)

;;------------
;; Various
;;-------------
(setq confirm-kill-emacs #'y-or-n-p)
(tool-bar-mode 0)
(global-flycheck-mode)

(setq sentence-end-double-space nil)
(show-paren-mode)

;;--------------
;; Help - transient-dwim
;;--------------
(which-key-mode 1)
(global-set-key (kbd "C-M-SPC") #'which-key-show-major-mode)
(global-set-key (kbd "M-=") 'transient-dwim-dispatch)

;; Helpful ---------------
(global-set-key (kbd "C-h f") #'helpful-callable)
(global-set-key (kbd "C-h v") #'helpful-variable)
(global-set-key (kbd "C-h k") #'helpful-key)
(global-set-key (kbd "C-h M-.") #'helpful-at-point)


;;------------
;; Ivy
;;----------
(ivy-mode 1)
(setq ivy-use-virtual-buffers t)
(setq ivy-count-format "(%d/%d) ")

;;-----------------
;; YASnippet
;;-----------------
(require 'yasnippet)
(yas-reload-all)
(yas-global-mode)

;;--------------
;;Projectile
;;-------------------
(require 'projectile)
(define-key projectile-mode-map (kbd "C-c p") 'projectile-command-map)
(projectile-mode +1)

;;------------------
;; Eglot
;;------------------
(require 'eglot)
;;use flychech instead of flymake
(add-to-list 'eglot-stay-out-of 'flymake)

;;suggested by the repo
(define-key eglot-mode-map (kbd "C-c r") 'eglot-rename)
(define-key eglot-mode-map (kbd "C-c o")
'eglot-code-action-organize-imports)
(define-key eglot-mode-map (kbd "C-c h") 'eldoc)

;;----------------
;; Language Server Protocol
;;--------------
(require 'lsp-mode)
;; apparently, this must be done before loading in any way lsp-mode
(setq lsp-keymap-prefix "C-c l")
(define-key lsp-mode-map (kbd "C-c l") lsp-command-map)
;; this must be called after to let which-key see the new prefix, think
(add-hook 'lsp-mode-hook 'lsp-enable-which-key-integration)
(define-key lsp-mode-map (kbd "M-RET") 'lsp-execute-code-action)
;;suggested by lsp-java doc
(setq lsp-completion-enable-additional-text-edit nil)

;;copied from https://emacs-lsp.github.io/lsp-mode/tutorials/CPP-guide/
;;probably is better
(with-eval-after-load 'lsp-mode
  (add-hook 'lsp-mode-hook #'lsp-enable-which-key-integration)
  (yas-global-mode))

;; lsp-mode is not activated by default in the c++-hook so that
;; the second bug can also be reproduced.
;; However, if needed, uncomment this line
;; (add-hook 'c++-mode-hook 'lsp)

;;---------------
;; lsp-ui
;;--------------
(eval-after-load 'lsp
  (eval-after-load 'lsp-ui
    (setq lsp-ui-sideline-show-hover t)
    ))

;;-------------------
;; Company (should be called after lsp-mode?)
;;------------------
(add-hook 'prog-mode-hook 'company-mode)
(define-key prog-mode-map (kbd "C-M-i") 'company-complete)
(eval-after-load 'comapny '(define-key company-mode-map (kbd "<C-tab>")
'company-complete))
(eval-after-load 'company '(define-key company-active-map (kbd "<tab>")
'company-complete))
(eval-after-load 'company '(add-to-list 'company-backends 'company-capf))
(setq company-minimum-prefix-length 1)

(run-hooks 'after-init-hook)

Re: Xref/tags/lsp possible bug

[Eli Zaretskii]

> From: Alessandro Bertulli <alessandro.bertulli96@gmail.com>
> Date: Sat, 23 Apr 2022 14:17:03 +0200
> 
> - First, when using LSP modes (both lsp-mode and eglot), the command
>   xref-find-definitions only acts on the symbol under point. I
>   explain: usually, in vanilla Emacs (emacs -q) if point is on a
>   whitespace/not on a symbol, invoking the command (bound to M-.) will
>   prompt you for an identifier to search for. However, when LSP is
>   active, it directly searches for an empty symbol, for example
>   reporting something similar to: "Symbol  not found" (not the double
>   space, which makes me think it searches for a symbol named ""). As I
>   said, this happend both for lsp-mode and eglot, so maybe is a
> problem of xref?
> 
> - Second, and minor: while I was investigating, I noticed that,
>   depending on the loaded packages, when giving M-., xref sometimes
>   asks for the symbol to search, sometimes it asks the TAGS table
> first. What package can be the problem?
> 
> Do you think it is a bug of Emacs? Do you know a way to resolve?
> Thanks to all!

Every major mode needs to set up its Xref support.  It sounds to me
that what you experience are the differences in the major modes you
use in each case.

Re: Xref/tags/lsp possible bug

[Alessandro Bertulli]

> Every major mode needs to set up its Xref support.  It sounds to me
> that what you experience are the differences in the major modes you
> use in each case.

Thanks, that makes sense. I'll try asking on the lsp-mode
support/mailing list. Thanks a lot, especially for the speed in
answering!

Re: Xref/tags/lsp possible bug

[Dmitry Gutov]

On 23.04.2022 15:52, Eli Zaretskii wrote:
> Every major mode needs to set up its Xref support.  It sounds to me
> that what you experience are the differences in the major modes you
> use in each case.

Not exactly: lsp-mode is not a major mode, nor does it include any major 
modes.

But it does indeed hook into Xref, and the support requests about the 
related behavior should be directed there.

I'll try to briefly answer the questions, though.

Re: Xref/tags/lsp possible bug

[Dmitry Gutov]

Hi!

On 23.04.2022 15:17, Alessandro Bertulli wrote:
> Hi all!
> 
> I just stumbled upon a very strange behavior of Emacs, I don't know if
> it is a bug or an error of my configuration. Basically, I noticed two
> things:
> 
> - First, when using LSP modes (both lsp-mode and eglot), the command
>    xref-find-definitions only acts on the symbol under point. I
>    explain: usually, in vanilla Emacs (emacs -q) if point is on a
>    whitespace/not on a symbol, invoking the command (bound to M-.) will
>    prompt you for an identifier to search for. However, when LSP is
>    active, it directly searches for an empty symbol, for example
>    reporting something similar to: "Symbol  not found" (not the double
>    space, which makes me think it searches for a symbol named ""). As I
>    said, this happend both for lsp-mode and eglot, so maybe is a
> problem of xref?

AFAIK it's a limitation of the LSP protocol: it's unable to perform 
completion on all symbol names globally. Nor is it able, in general, to 
find a symbol by name (it needs a location of an existing reference in 
some file). So that affects how lsp-mode's Xref integration works (and 
Eglot's).

> - Second, and minor: while I was investigating, I noticed that,
>    depending on the loaded packages, when giving M-., xref sometimes
>    asks for the symbol to search, sometimes it asks the TAGS table
> first. What package can be the problem?

ivy-mode, probably. Though it's not necessarily a "problem".

ivy-mode takes over the completing-read UI, and it tries to show the 
available completions right away. The identifier completion table 
provided by the default Xref backend uses tags, and when the table is 
not loaded, has to prompt for the file to load it from.

Re: Xref/tags/lsp possible bug

[Alessandro Bertulli]

Thanks Dmitry!

> AFAIK it's a limitation of the LSP protocol: it's unable to perform
> completion on all symbol names globally. Nor is it able, in general, to
> find a symbol by name (it needs a location of an existing reference in
> some file). So that affects how lsp-mode's Xref integration works (and
> Eglot's).

Indeed, you're right, I opened an issue on lsp-mode's GitHub and the
maintainer (Ivan) kindly answered me. I report here its answer, in case
it is useful to someone:

"In lsp protocol, there is no convenient method to list the symbols.
We do some hacks using document symbols to provide some handling when
the method is invoked like that C-u M-x xref-find-definitions(or with
(setq xref-prompt-for-identifier t)) but that is not really what the
xref wants. This feature of xref is designed with tags in mind and it
does not fit well with lsp protocol. Thus we have created lsp-find-*
set of functions to make the difference between lsp and xref more
evident."

Okay, it's not a real problem, but I wonder why is it, though. I
suppose LSP does have a sort of symbol table kept in memory, so I
think it wouldn't be a big problem to allow the user to query it. I
suppose it's something that can be added in a future specification.

As a side note, reading your answer to Eli: yes, I got the hint it
could be a problem of Xref, that's a part of why I asked here (being
Xref a part of Emacs, or at least GNU Elpa).

Anyway, thank you all for your support!

Alessandro

Re: Xref/tags/lsp possible bug

[Stefan Monnier via Users list for the GNU Emacs text editor]

> Okay, it's not a real problem, but I wonder why is it, though.
> I suppose LSP does have a sort of symbol table kept in memory, so
> I think it wouldn't be a big problem to allow the user to query it.
> I suppose it's something that can be added in a future specification.

I must say I don't know why that is, but FWIW in my local experimental
language (Typer) it makes some sense: Typer has no top-level
environment, so if a file is defined as:

   fun1 x = <foo>;
   fun2 y = <bar>;
   ...

it's really the same as if it was:

   let fun1 x = <foo>
   in let fun2 y = <bar>;
      in ...

You can still try to cobble up some kind of "list of top-levelish
definitions" but it's inevitably a hack.

OTOH, LSP could define the set of possible choices as the set of
possible "breadcrumbs" info.


        Stefan

Re: Xref/tags/lsp possible bug

[Dmitry Gutov]

On 25.04.2022 23:05, Stefan Monnier via Users list for the GNU Emacs 
text editor wrote:
> You can still try to cobble up some kind of "list of top-levelish
> definitions" but it's inevitably a hack.

Yes, in most languages there will be at least some symbols that have no 
meaningful "global" names.

LSP could provide a table of all global symbols, though. And it kind of 
does, but not exactly. Not in a way that's useful to build an Emacs 
completion table from. But it will hopefully improve the spec further.

As for the reason why: my guess is that the protocol was designed as a 
common ground for a number of existing code assistance libraries/tools 
already developed within Microsoft, rather than something from a blank 
slate, and those are inevitably biased toward a particular UI.

Re: Xref/tags/lsp possible bug

[Stefan Monnier]

> As for the reason why: my guess is that the protocol was designed as
> a common ground for a number of existing code assistance libraries/tools
> already developed within Microsoft, rather than something from a blank
> slate, and those are inevitably biased toward a particular UI.

Indeed, the protocol has a fairly strong "ad-hoc" feel to it.
It doesn't cleanly separate the UI from the rest.  For example, a lot of
presentation choices are actually made in the server rather than in
the client.

It's also reflected in the name of requests like "hover" which
fundamentally asks for information related to some position in the code,
but encodes the idea that this information will be presented as popup
boxes as the mouse "hovers" over that location.


        Stefan

Re: Xref/tags/lsp possible bug

[Dmitry Gutov]

On 26.04.2022 00:48, Stefan Monnier wrote:
> Indeed, the protocol has a fairly strong "ad-hoc" feel to it.
> It doesn't cleanly separate the UI from the rest.  For example, a lot of
> presentation choices are actually made in the server rather than in
> the client.

Yep.

> It's also reflected in the name of requests like "hover" which
> fundamentally asks for information related to some position in the code,
> but encodes the idea that this information will be presented as popup
> boxes as the mouse "hovers" over that location.

...and doesn't otherwise describe what that data should contain.