bug#61350: Eglot over Tramp freezes with large project

[Michael Albinus <michael.albinus@gmx.de>]

Thomas Koch <thomas@koch.ro> writes:

Hi Thomas,

> I've reproduced the problem with Emacs 27.1 from Debian and a minimal
> Emacs 28.2. installed via nixpkg. The follwing describes a hopefully
> reliable reproduction.
>
> 1. Have a client system with Emacs + eglot and an SSH server.
>
> 2. On the server:
>
>   git clone --depth 1 --no-tags --single-branch \
>       -b eglot-tramp-freeze-repro \
>       https://github.com/thkoch2001/yacy_search_server
>
> 3. On the server download and untar
>    https://download.eclipse.org/jdtls/milestones/1.19.0/jdt-language-server-1.19.0-202301171536.tar.gz
>
> 4. Still on server make a symlink from ~/bin/jdtls to bin/jdtls from the
> tarball.
>
> 5. Install eglot and use this emacs config on client:
>
> (custom-set-variables
>  '(tramp-remote-path
>    '("/run/current-system/sw/bin" "/bin" "/usr/bin" "/sbin" "/usr/sbin" "/usr/local/bin" "/usr/local/sbin" "/local/bin" tramp-own-remote-path))
>  '(tramp-use-ssh-controlmaster-options nil)
> ;; problem exists also without previous line
> )
>
> 6. Open source/net/yacy/yacy.java over Tramp with ssh:
>
> 7. M-x eglot

Thanks for the concise recipe. Since I have used a virgin Ubuntu VM as
test target, one additional step is needed: Install Java.

> Freeze should happen after a few seconds.

Yep, I could reproduce it with both Emacs 29.0.60 and 30.0.50. And the
backtrace is

--8<---------------cut here---------------start------------->8---
  backtrace()
  tramp-error((tramp-file-name "ssh" nil nil "ubuntu-2204" nil "/home/albinus/src/yacy_search_server/examples/Simp..." nil) quit "")
  tramp-signal-hook-function(quit nil)
  accept-process-output(#<process *tramp/ssh ubuntu-2204*> nil nil t)
  tramp-accept-process-output(#<process *tramp/ssh ubuntu-2204*>)
  tramp-wait-for-regexp(#<process *tramp/ssh ubuntu-2204*> nil "\\(?:^\\|\0\\)\\(?:[^\n#$]*///3f62f1d55bd34cba7fe730c633...")
  tramp-wait-for-output(#<process *tramp/ssh ubuntu-2204*>)
  tramp-send-command((tramp-file-name "ssh" nil nil "ubuntu-2204" nil "/home/albinus/src/yacy_search_server/examples/Simp..." nil) "\\readlink --canonicalize-missing /home/albinus/src...")
  tramp-send-command-and-check((tramp-file-name "ssh" nil nil "ubuntu-2204" nil "/home/albinus/src/yacy_search_server/examples/Simp..." nil) "\\readlink --canonicalize-missing /home/albinus/src...")
  tramp-sh-handle-file-truename("/ssh:ubuntu-2204:/home/albinus/src/yacy_search_ser...")
  apply(tramp-sh-handle-file-truename "/ssh:ubuntu-2204:/home/albinus/src/yacy_search_ser...")
  tramp-sh-file-name-handler(file-truename "/ssh:ubuntu-2204:/home/albinus/src/yacy_search_ser...")
  apply(tramp-sh-file-name-handler file-truename "/ssh:ubuntu-2204:/home/albinus/src/yacy_search_ser...")
  tramp-file-name-handler(file-truename "/ssh:ubuntu-2204:/home/albinus/src/yacy_search_ser...")
  file-truename("/ssh:ubuntu-2204:/home/albinus/src/yacy_search_ser...")
  find-buffer-visiting("/ssh:ubuntu-2204:/home/albinus/src/yacy_search_ser...")
  #f(compiled-function (arg1 arg2 &rest rest) "Handle notification publishDiagnostics." #<bytecode 0x5bbf29480580234>)(#<eglot-lsp-server eglot-lsp-server-bfebda> textDocument/publishDiagnostics :uri "file:///home/albinus/src/yacy_search_server/exampl..." :diagnostics [])
  apply(#f(compiled-function (arg1 arg2 &rest rest) "Handle notification publishDiagnostics." #<bytecode 0x5bbf29480580234>) #<eglot-lsp-server eglot-lsp-server-bfebda> textDocument/publishDiagnostics (:uri "file:///home/albinus/src/yacy_search_server/exampl..." :diagnostics []))
  eglot-handle-notification(#<eglot-lsp-server eglot-lsp-server-bfebda> textDocument/publishDiagnostics :uri "file:///home/albinus/src/yacy_search_server/exampl..." :diagnostics [])
  apply(eglot-handle-notification #<eglot-lsp-server eglot-lsp-server-bfebda> textDocument/publishDiagnostics (:uri "file:///home/albinus/src/yacy_search_server/exampl..." :diagnostics []))
  #f(compiled-function (server method params) #<bytecode -0xa39f6ed8241f361>)(#<eglot-lsp-server eglot-lsp-server-bfebda> textDocument/publishDiagnostics (:uri "file:///home/albinus/src/yacy_search_server/exampl..." :diagnostics []))
  jsonrpc-connection-receive(#<eglot-lsp-server eglot-lsp-server-bfebda> (:jsonrpc "2.0" :method "textDocument/publishDiagnostics" :params (:uri "file:///home/albinus/src/yacy_search_server/exampl..." :diagnostics [])))
  jsonrpc--process-filter(#<process EGLOT (yacy_search_server/(java-mode java-ts-mode))> "pc\":\"2.0\",\"method\":\"$/progress\",\"params\":{\"token\":...")
--8<---------------cut here---------------end--------------->8---

The problem is clear. We are in jsonrpc--process-filter (accepting process
output). This runs eglot-handle-notification, and that runs
file-truename. The latter needs to check remote, so Tramp sends a remote
command, and waits for that output (calling accept-process-output again).

Since jsonrpc always accepts output from *all* running processes, there
could be (and is!) the constellation, that process output has been read
already, and Tramp didn't get it, waiting for the output forever. I
could mitigate the problem partly by changing the arguments of
accept-process-output in jsonrpc-request, and eglot could work
successfully, sometimes. And sometimes not. And even then, I ran into
the famous "Forbidden reentrant call" error, see bug#60534.

I will continue to investigate, but after playing with this for some
days, I feel we must add threads to Tramp, when there are several
concurrent processes. I've tried this already some years agao, and I
failed. Maybe it is time now for a new attempt.

Best regards, Michael.