bug#61350: Eglot over Tramp freezes with large project

[Michael Albinus <michael.albinus@gmx.de>]

João Távora <joaotavora@gmail.com> writes:

Hi João,

>> is rather, that Tramp must know where the output to be parsed starts in
>> the buffer.
>
> Right, and this is where 'point' and 'process-mark' come in.  Process
> mark is where the internal filter last left the insertion, and point is
> where you the programmer last left your parsing.

Yes, but Tramp doesn't get the whole output with one
accept-process-output call. It must use a while loop, until everything
it expects has been arrived. So the process-mark also moves again and
again, and cannot be used by Tramp directly.

>> If another process has consumed the output, even if it is pushed into
>> the correct buffer, Tramp doesn't know.
>
> Why?  May I ask -- perhaps naively -- why can't you can't just
>
>   (with-current-buffer proc (= (point) (process-mark)))
>
> to "know"?

See above.

> In my experience, something like this is usually sufficient.  One parses
> the buffer for incoming messages looking for regexps, magic byte
> sequences, etc.  One always leaves point after a successful search
> (search-forward-regexp has this behaviour and it's very convenient).
> Eventually, point may be left exactly at process-mark, or not, depending
> on how much data came in, a full message, multiple full messages, or
> some fractional message.
>
> Regardless, next time you want to get messages from the process, you
> perform a check before you go on a potentially blocking call to fetch
> more output.  The check is usually "has process-mark advanced behind my
> back from other libraries I don't control?"  Here, jsonrpc.el's a-p-o
> calls are the "behing your back".  After checking, you know if you have
> enough data to form a full message, or if you need to go on a
> potentially blocking a-p-o call.

Everything correct. But the problem was that not the whole expected
output hasn't arrived the buffer. I thought it was "stolen" by another
process. But yesterday's debugging has shown, that ssh ControlMaster
seems to be guilty; it cannot handle large amount of data reliably.

> But somehow I suspect you know all this by heart already!!  In fact,
> from the backtrace you posted Fri, 17 Feb 2023, it's clear the hang
> happend in 'tramp-wait-for-regexp' whic starts 
>
> (defun tramp-wait-for-regexp (proc timeout regexp)
>   ...
>   (let ((found (tramp-check-for-regexp proc regexp)))
>     (cond (timeout
> 	   (with-timeout (timeout)
> 	     (while (not found)
> 	       (tramp-accept-process-output proc)
>
> Here, exactly how I imaged, you first check for "found" before venturing
> into the blocking tramp-accept-process-output call.  So it looks to me
> that you're doing conceptually the right thing, but it's
> tramp-check-for-regexp who is missing the fact that there is a perfectly
> good message in process buffer already.  At least according to what I
> understood from your account of the problem.

Yes, that's the place. Tramp waits for a trailing shell prompt that it
tells that everything has been sent from remote. But this shell prompt
didn't arrive, and Tramp hangs in waiting for this.

>> Honestly, I still don't understand really what's up. Let's see whether
>> adding threads to Tramp helps.
>
> I'll try to setup a VM myself with the reproduction recipe that Thomas
> used.  I'm reasonably confident that two process-based extensions such
> as Jsonrpc.el and TRAMP can coeexist if each follows process etiquette.

They will cooperate, definitely!

> João

Best regards, Michael.