From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: =?UTF-8?Q?Jo=C3=A3o_?= =?UTF-8?Q?T=C3=A1vora?= Newsgroups: gmane.emacs.bugs Subject: bug#61350: Eglot over Tramp freezes with large project Date: Sat, 25 Feb 2023 23:09:01 +0000 Message-ID: <878rglxrzm.fsf@gmail.com> References: <87y1ootw2t.fsf@gmail.com> <69968923.705640.1677163650760@office.mailbox.org> <87a613f0b7.fsf@gmx.de> <87r0udvmzr.fsf@gmx.de> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="17869"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Gnus/5.13 (Gnus v5.13) Cc: Thomas Koch , 61350@debbugs.gnu.org To: Michael Albinus Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Sun Feb 26 00:08:26 2023 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1pW3eU-0004Tg-MW for geb-bug-gnu-emacs@m.gmane-mx.org; Sun, 26 Feb 2023 00:08:26 +0100 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pW3e8-0004G7-4y; Sat, 25 Feb 2023 18:08:04 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pW3e6-0004Fz-N2 for bug-gnu-emacs@gnu.org; Sat, 25 Feb 2023 18:08:02 -0500 Original-Received: from debbugs.gnu.org ([209.51.188.43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pW3e6-0007qS-6T for bug-gnu-emacs@gnu.org; Sat, 25 Feb 2023 18:08:02 -0500 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1pW3e5-0002SH-Kc for bug-gnu-emacs@gnu.org; Sat, 25 Feb 2023 18:08:01 -0500 X-Loop: help-debbugs@gnu.org Resent-From: =?UTF-8?Q?Jo=C3=A3o_?= =?UTF-8?Q?T=C3=A1vora?= Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Sat, 25 Feb 2023 23:08:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 61350 X-GNU-PR-Package: emacs Original-Received: via spool by 61350-submit@debbugs.gnu.org id=B61350.16773664389380 (code B ref 61350); Sat, 25 Feb 2023 23:08:01 +0000 Original-Received: (at 61350) by debbugs.gnu.org; 25 Feb 2023 23:07:18 +0000 Original-Received: from localhost ([127.0.0.1]:41808 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pW3dN-0002RE-MK for submit@debbugs.gnu.org; Sat, 25 Feb 2023 18:07:18 -0500 Original-Received: from mail-wr1-f49.google.com ([209.85.221.49]:46825) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pW3dM-0002R2-D6 for 61350@debbugs.gnu.org; Sat, 25 Feb 2023 18:07:17 -0500 Original-Received: by mail-wr1-f49.google.com with SMTP id bw19so2636060wrb.13 for <61350@debbugs.gnu.org>; Sat, 25 Feb 2023 15:07:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1677366430; h=content-transfer-encoding:mime-version:user-agent:message-id:date :references:in-reply-to:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4vqFjOwoIAWNuCI43sXv3cK4Oyvqv5OZylCUiGJgj6M=; b=RzYg3zhv1nxuomZjIskw/TLp6MyhUCgM5XMKTUHAlUCfhmOveL7nJV29SxDGwWMs3U CVl0qgvKoYASOQoT48i6UwegdfUWd1GXi0qKM6os6fv/qjX4BUf4eXvQXNf4Lcaf+tMj 6PJiOYITc+QaZfHfkSq82S3OUoWlKnCsgYa0GngMwX0zEE+7Fch7VP5Z4H7+q1jVXn/N hFMIyY7uPy4YawEqku6bU1sCzf1lQJjlSJiQR5q6Qf8xgMRqeK3ZfF1YeXvQoWRI5yGs 0h6DVtILBrPQlasJ1870EN6/RD96qsZmd2HttGnRy8bwS8ngPYqbxfFKRa3RZKzarB0C CAeg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1677366430; h=content-transfer-encoding:mime-version:user-agent:message-id:date :references:in-reply-to:subject:cc:to:from:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=4vqFjOwoIAWNuCI43sXv3cK4Oyvqv5OZylCUiGJgj6M=; b=nS7wzvnlWC6T4jkEkzvqEzFgBP/XElM26Oxzw8OxO4M788zE6WRP/rxQjE4vT4iU8F LXbqCbg/tKAUJ17wpnvRtwZijyVCS+mcRs5IiuI9SOzub/dIu6XQNZsl8ah7N0rftOtv bJH5JnIt+K0EK03OOK6hjmnFzwzCSR3seUhHdWFjZXbHIzXTVeHoh9YMp73YzxI5UmK1 BBMIa+mL02Yv3NpofdvqZzzrLkrxCR10cmeZTqqd2xYx0FNPmxqivaKO5ug8h5+eJfuT EzCk5VW6WdUCCZe9hkvbS9DgLyYnOB4fjJHSiiFXE3f+kmzASPAULBgrFW+K7DCuydrs Q8Fw== X-Gm-Message-State: AO0yUKWV0I0zR03EgA/wN62rA3Y5ekNXyMaBYwUQz3Fh8V3M6J6llesS vzzwvx+czjTv4UpVGqhm1yd3S5BYk7I= X-Google-Smtp-Source: AK7set8V2qMEgchwHT33YN+lz/k72KYwZtQlhSFkZbBoOAj4KR0YXR95O4OFV3ObRjHuadnbXqYxeg== X-Received: by 2002:a5d:4c83:0:b0:2c9:12dc:260 with SMTP id z3-20020a5d4c83000000b002c912dc0260mr2372779wrs.33.1677366430122; Sat, 25 Feb 2023 15:07:10 -0800 (PST) Original-Received: from krug (87-196-72-142.net.novis.pt. [87.196.72.142]) by smtp.gmail.com with ESMTPSA id l17-20020a5d6691000000b002c54e26bca5sm2893147wru.49.2023.02.25.15.07.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 25 Feb 2023 15:07:08 -0800 (PST) In-Reply-To: <87r0udvmzr.fsf@gmx.de> (Michael Albinus's message of "Sat, 25 Feb 2023 15:27:36 +0100") X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.bugs:256742 Archived-At: Michael Albinus writes: > Hi Jo=C3=A3o, > >>> You don't call jsonrpc--notification-dispatcher anymore in the >>> process filter directly. But calling it by a timer has the same effect: >>> The process filter (accepting output) is still active, >> >> What exactly do you mean by "still active" and what process are you >> referring to? Jsonrpc's or Tramp? > > Your timer is called immediately, while you're still in > jsonrpc--process-filter I believe. I don't think so. Any timer function runs in its stack, after the stack where jsonrpc--process-filter happened has unwound. But you've answered my question: by "active" you mean "under the stack frame of jsonrpc--process-filter". >>> > Since jsonrpc always accepts output from *all* running processes, the= re >>> > could be (and is!) the constellation, that process output has been re= ad >>> > already, and Tramp didn't get it, waiting for the output forever. >> >> I could understand if we were talking C and read() here, but the process >> output of other processes read by jsonrpc's call to accept-process-output >> surely must have run through some filter function, it wasn't just lost >> AFAIK. You've probably seen this trick a mililon times, but markers >> in the process buffer is what is used commonly. > > It wasn't lost. The process output was retrieved and placed into the > Tramp buffer, w/o Tramp's interaction. That's great and quite normal. > Tramp doesn't use a process filter for its own connections. Every process in Emacs has a process filter, though it might not be a custom process filter. If you don't give it one, it is assigned internal-default-process-filter, which as the docstring explains, simply inserts process output into the buffer (of course you probably know this, I'm just stating for other readers). > is rather, that Tramp must know where the output to be parsed starts in > the buffer. Right, and this is where 'point' and 'process-mark' come in. Process mark is where the internal filter last left the insertion, and point is where you the programmer last left your parsing. > If another process has consumed the output, even if it is pushed into > the correct buffer, Tramp doesn't know. Why? May I ask -- perhaps naively -- why can't you can't just (with-current-buffer proc (=3D (point) (process-mark))) to "know"? In my experience, something like this is usually sufficient. One parses the buffer for incoming messages looking for regexps, magic byte sequences, etc. One always leaves point after a successful search (search-forward-regexp has this behaviour and it's very convenient). Eventually, point may be left exactly at process-mark, or not, depending on how much data came in, a full message, multiple full messages, or some fractional message. Regardless, next time you want to get messages from the process, you perform a check before you go on a potentially blocking call to fetch more output. The check is usually "has process-mark advanced behind my back from other libraries I don't control?" Here, jsonrpc.el's a-p-o calls are the "behing your back". After checking, you know if you have enough data to form a full message, or if you need to go on a potentially blocking a-p-o call. But somehow I suspect you know all this by heart already!! In fact, from the backtrace you posted Fri, 17 Feb 2023, it's clear the hang happend in 'tramp-wait-for-regexp' whic starts=20 (defun tramp-wait-for-regexp (proc timeout regexp) ... (let ((found (tramp-check-for-regexp proc regexp))) (cond (timeout (with-timeout (timeout) (while (not found) (tramp-accept-process-output proc) Here, exactly how I imaged, you first check for "found" before venturing into the blocking tramp-accept-process-output call. So it looks to me that you're doing conceptually the right thing, but it's tramp-check-for-regexp who is missing the fact that there is a perfectly good message in process buffer already. At least according to what I understood from your account of the problem. So my suspicion is in tramp-check-for-regexp. I found it a bit hard to read to find an obvious culprit, and I still haven't a working setup to reproduce this... >>> Again, I still believe we need a general solution in Tramp, using threa= ds. >> >> I don't understand what is conceptually impossible (or very hard) to >> achieve with evented IO like we have in Emacs. > > I've tried different patches, mainly in tramp-accept-process-output. It > improves the situation a little bit, but not reliably. Sometimes the > test works, sometimes it blocks. And even if it doesn't block, a while > later we run into the "Forbidden reentrant call of Tramp" error. I had recently a problem with reentrant calls in jsonrpc--process-filter. This is why you find a run-with-timer call there, right at the beginning. This removes the reentrancy because as written before, timers run in their own stack. This was the fix to a nasty Eglot bug#60088 with similar "hanging" behaviour. > Honestly, I still don't understand really what's up. Let's see whether > adding threads to Tramp helps. I'll try to setup a VM myself with the reproduction recipe that Thomas used. I'm reasonably confident that two process-based extensions such as Jsonrpc.el and TRAMP can coeexist if each follows process etiquette. Jo=C3=A3o