bug#61350: Eglot over Tramp freezes with large project

[Michael Albinus <michael.albinus@gmx.de>]

Thomas Koch <thomas@koch.ro> writes:

Hi Thomas,

> My current working theory is:
>
> 1. There is some buffer in SSH (or TCP) that gets filled by the language server sending data to eglot.
>
> 2. Tramp sends a command over SSH.
>
> 3. Tramp sets the JUST-THIS-ONE option of accept-process-output to t since https://debbugs.gnu.org/12145.
>
> 4. The response to 2. can not arrive due to the buffer being filled in 1. Tramp blocks the emacs main thread.

It is a little bit more specific. Tramp speaks with the remote shell in
order to execute commands like '(file-truename "...")'. This
is transformed by Tramp into the shell command

--8<---------------cut here---------------start------------->8---
\readlink --canonicalize-missing /home/albinus/src/yacy_search_server/examples/SimpleSearchClient/pom.xml 2>/dev/null; echo tramp_exit_status $?
--8<---------------cut here---------------end--------------->8---

Tramp expects the result of the readlink call, and the return value (in
order to know whether this result can be used). And it expects a special
shell prompt, which is the indication, that the remote command has been
finished. So it calls tramp-accept-process-output, unitl the connection
buffer contains something like

--8<---------------cut here---------------start------------->8---
/home/albinus/src/yacy_search_server/examples/SimpleSearchClient/pom.xml
tramp_exit_status 0
///4b3b7d4fa561141e84c94a1cf25e8575#$
--8<---------------cut here---------------end--------------->8---

This special shell prompt is initialized at process startup, by setting
PS1.

This works fine, until there is the concurrency with the Eglot
process. According to my debugging, Tramp sees only part of the
output. The final shell prompt, ///4b3b7d4fa561141e84c94a1cf25e8575#$,
doesn't arrive, and so Tramp is blocked.

> I tested my theory by deleting (and thus disabling) the JUST-THIS-ONE
> argument in all calls to accept-process-output in tramp.el and
> tramp-sh.el. Eglot did not freeze anymore in two tests with two
> different systems (but the same jdtls binary on the same Debian
> version).

Yes. But I still don't understand what's the difference in scenario.

> In one test however I saw this in the message buffer:
>
> [jsonrpc] (warning) Invalid JSON: (control character 0xd <callback> 1
> 381 381)
> {"jsonrpc":"2.0","method":"textDocument/publishDiagnostics","params":{"uri":"file:///home/thk/git/yacy_search_server/source/net/yacy/htroot/ConfigSearchBox.java","diagnostics":[{"range":{"start":{"line":35,"character":86},"end":{"line":35,"character":94}},"severity":3,"code":"1102","source":"Java","message":"At
> least one of the problems in category \u0027unuseContent-Length: 798
>
> {"jsonrpc":"2.0","method":"textDocument/publ
>
> I already sent the above to Michael in an out-of-band thread (thanks
> for your patience!).

I suspect this is due to the "Forbidden reentrant call", but it isn't obvious.

> I have a vague feeling, that Tramp could be improved with a work queue
> such that requests to tramp from notification or timer threads get
> blocked while another tramp command is still waiting for a
> reply.

Exactly. This is what I want to achieve by using threads for protecting
the communication with the remote processes. Or at least with the main
connection process.

> Additionally I understand that Joao has an idea to use markers
> controlled by Tramp. - I'm sorry that I can not (yet) contribute with
> putting both ideas into code.
>
> There are two statements in this bug thread that I don't yet understand and that might be worth more elaboration. I volunteer as a debugging rubber duck[1]. However if you both understand each other then never mind:
>
> [1] https://en.wikipedia.org/wiki/Rubber_duck_debugging
>
> Joao: "markers in the process buffer is what is used commonly"
>
> I understand that tramp sends only one command at a time per
> connection. Otherwise the reentrant error is thrown. The command
> result gets written to a connection-specific buffer. The output is
> parsed from point-min and the buffer content deleted after parsing and
> before sending the next command. So what should there be improved?
> Jsonrpc needs its own markers because N messages can arrive at any
> given time and the buffer can not be deleted after each message.

Meanwhile I don't believe any longer that markers are the problem.

> Michael: "If another process has consumed the output, even if it is pushed into the correct buffer, Tramp doesn't know."
>
> What exact scenario should that be? Emacs writes the output (if
> there's no filter) in the correct buffer for the process. Tramp might
> call accept-process-output unnecessarily because the output is already
> in the buffer due to a accept-process-output call of some other
> code. But in this case Tramp will find the output there eventually,
> maybe after having waited until a timeout. This could be improved by
> checking for already present output before calling
> accept-process-output?

I stand corrected. The problem isn't that another process has consumed
the output, but rather that not everything arrives. For whatever reason.

Best regards, Michael.