From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: =?UTF-8?Q?Jo=C3=A3o_?= =?UTF-8?Q?T=C3=A1vora?= Newsgroups: gmane.emacs.bugs Subject: bug#61350: Eglot over Tramp freezes with large project Date: Wed, 01 Mar 2023 14:10:47 +0000 Message-ID: <87ilfkh89k.fsf@gmail.com> References: <87y1ootw2t.fsf@gmail.com> <69968923.705640.1677163650760@office.mailbox.org> <87a613f0b7.fsf@gmx.de> <87r0udvmzr.fsf@gmx.de> <878rglxrzm.fsf@gmail.com> <87cz5wmjbx.fsf@gmx.de> <87h6v8f7u9.fsf@gmail.com> <87o7pflfcd.fsf@gmx.de> <87wn43e9ht.fsf@gmail.com> <874jr6oont.fsf@gmx.de> <87sfeqd4zi.fsf@gmail.com> <877cw1swjm.fsf@gmx.de> <87k0016dgo.fsf@gmx.de> <1458446553.50372.1677606917251@office.mailbox.org> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="36381"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Gnus/5.13 (Gnus v5.13) Cc: Michael Albinus , 61350@debbugs.gnu.org To: Thomas Koch Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Wed Mar 01 15:11:33 2023 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1pXNB6-0009Lb-K0 for geb-bug-gnu-emacs@m.gmane-mx.org; Wed, 01 Mar 2023 15:11:32 +0100 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pXN9i-0008KV-0N; Wed, 01 Mar 2023 09:10:06 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pXN9e-0007wT-Rs for bug-gnu-emacs@gnu.org; Wed, 01 Mar 2023 09:10:03 -0500 Original-Received: from debbugs.gnu.org ([209.51.188.43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pXN9e-0006DG-Bz for bug-gnu-emacs@gnu.org; Wed, 01 Mar 2023 09:10:02 -0500 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1pXN9e-0002rD-1p for bug-gnu-emacs@gnu.org; Wed, 01 Mar 2023 09:10:02 -0500 X-Loop: help-debbugs@gnu.org Resent-From: =?UTF-8?Q?Jo=C3=A3o_?= =?UTF-8?Q?T=C3=A1vora?= Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Wed, 01 Mar 2023 14:10:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 61350 X-GNU-PR-Package: emacs Original-Received: via spool by 61350-submit@debbugs.gnu.org id=B61350.167767974310882 (code B ref 61350); Wed, 01 Mar 2023 14:10:02 +0000 Original-Received: (at 61350) by debbugs.gnu.org; 1 Mar 2023 14:09:03 +0000 Original-Received: from localhost ([127.0.0.1]:53068 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pXN8g-0002pD-Hb for submit@debbugs.gnu.org; Wed, 01 Mar 2023 09:09:03 -0500 Original-Received: from mail-wr1-f44.google.com ([209.85.221.44]:44546) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1pXN8e-0002op-Ho for 61350@debbugs.gnu.org; Wed, 01 Mar 2023 09:09:01 -0500 Original-Received: by mail-wr1-f44.google.com with SMTP id bx12so10184615wrb.11 for <61350@debbugs.gnu.org>; Wed, 01 Mar 2023 06:09:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1677679734; h=content-transfer-encoding:mime-version:user-agent:message-id:date :references:in-reply-to:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=1jnvB+ZgXnIUlQPSxXJeYui2yCWW3LPJyZHexy5V7L8=; b=kWzNY2bPZotPwdGjriUGMCZZalatdsvZcmOeWZAhskhLMDRAz9z4X4GtMM8Ec2M8wV pLokkKj0gQW//W05xXZuOPE2L2MqHHof+aBI3u+kX7iIRP0kjD0SBwGyVE2BfvQ+s4pi SFufhTSb6UJdpzHnvTUUXr8BViP7vO8acKbpNX2Up9vAY/m8qhX4Aodw4GJVvirbf3Yt 47tIM74qSvlsohX05u9dRLqN77Yh16w3IoT9ImAFYmZI0DxhgUsz9PgcHq2OKjfvEc5T G1Pl5/Cz/Zvi2p/ohK6jABib9fkAyBlKpPMf3C7pb9FJnNT8lJQq0lnf8Qoa1ZLnEyPH cHMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1677679734; h=content-transfer-encoding:mime-version:user-agent:message-id:date :references:in-reply-to:subject:cc:to:from:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=1jnvB+ZgXnIUlQPSxXJeYui2yCWW3LPJyZHexy5V7L8=; b=6aLlehVlKw6kcl08treusXJe/2gOlBO4JbwVc/a77WiHsjy7aADWS0fYl+wej2b0P7 CGpV6Qttr3RgjRJHOPFv8yRmfhaoipSH0t1zKhc4yCZvQrqhf2CSsRIbHk3eFIAQr6i4 DrJsjRpkgd8CLln0FL/bg/sGijC2CMepHOqSCEi5YvmVB87wsOwDoS8Zs5ZfuTWX65CJ Kyj1HzRboqa3xypvXjY/F44jrygJNvwPoGPg6mfchb3MD3DaiEEoWMZXmpxe4UD00mYt KIw7NjPA04JTHkOs0TRtRJl0N+UUuQ5wTQ8n372iDkhhjntam4lcQNfwnc8TjVjDHDxY uD8g== X-Gm-Message-State: AO0yUKX/OzDFvAhSuE5JD804LHHrfkhZVIBEseg3gQ2+FdvevXYdLrUc 9SnrPJKgeYJf6xXSAQ2vJvt1HzbTA70= X-Google-Smtp-Source: AK7set9zy9GtCRMab009BmV2ga+XoqLPUI4nENKFSh9mlIEwY0erInqJgW4fXtb3qFbtHuJonlKTHA== X-Received: by 2002:a5d:6041:0:b0:2c8:c667:1bb4 with SMTP id j1-20020a5d6041000000b002c8c6671bb4mr4744831wrt.48.1677679733973; Wed, 01 Mar 2023 06:08:53 -0800 (PST) Original-Received: from krug (87-196-72-142.net.novis.pt. [87.196.72.142]) by smtp.gmail.com with ESMTPSA id r14-20020adff70e000000b002c567881dbcsm12748450wrp.48.2023.03.01.06.08.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Mar 2023 06:08:53 -0800 (PST) In-Reply-To: <1458446553.50372.1677606917251@office.mailbox.org> (Thomas Koch's message of "Tue, 28 Feb 2023 19:55:17 +0200 (EET)") X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.bugs:257065 Archived-At: Thomas Koch writes: > Thank you both for being so constructively engaged in the bug I opened. > > I'll give Joao's patch a try tomorrow. (I'm too tired now.) However I don= 't see ControlMaster as the root cause at the moment. > > My current working theory is: > > 1. There is some buffer in SSH (or TCP) that gets filled by the language = server sending data to eglot. > 2. Tramp sends a command over SSH. > 3. Tramp sets the JUST-THIS-ONE option of accept-process-output to t sinc= e https://debbugs.gnu.org/12145. > 4. The response to 2. can not arrive due to the buffer being filled in 1.= Tramp blocks the emacs main thread. > I tested my theory by deleting (and thus disabling) the JUST-THIS-ONE > argument in all calls to accept-process-output in tramp.el and > tramp-sh.el. Eglot did not freeze anymore I don't think your observation necessarily validates your theory. If in '1.' you mean _an Emacs buffer_, the buffer in question is populated by Tramp's process filter (the default process filter, that is). Then accept-process-output with non-nil JUST-THIS-ONE would not prevent that buffer from getting the all the output, becasue the "THIS-ONE" is the process. But JUST-THIS-ONE _is_ relevant when there is more than one process. Here, there is. There's one process, the jsonrpc.el process, henceforth 'jprocess', and the Tramp process, henceforth 'tprocess'. jprocess receives only JSONRPC data from the LSP server. It "thinks" it is talking directly to a JSONRPC server, but in Tramp scenarios it is being fed data from tprocess, which is the process connected to the remote host. In tprocess, other things, such as shell interactions are going on. Michael can probably confirm, correct or deny this. When one (accept-process-output tprocess nil nil 'JUST-THIS-ONE=3Dt) one must be absolutely sure that tprocess is going to send _something_ "soon". If it doesn't, we'll hang indefinitely (until the process dies or the user quits) That's what has been confirmed through a backtrace. It's a particular accept-process-output call in tramp-wait-for-regexp that hangs, understandibly so. The need to ensure safety of that call is why there is a prior for any 'found' messages in 'tramp-wait-for-regexp'. Only then is the "risky" 'accept-process-output' attempted. (defun tramp-wait-for-regexp (proc timeout regexp) ... (let ((found (tramp-check-for-regexp proc regexp))) (cond (...) (t (while (not found) (tramp-accept-process-output proc) ;; WE "HANG" HERE SOMETIMES (unless (process-live-p proc) (tramp-error-with-buffer nil proc 'file-error "Process has died")) (setq found (tramp-check-for-regexp proc regexp))))) 'found' relies on searching the contents of what is already in tprocess's buffer, which tprocess's filter dumped there. So far so good. Now, 'tramp-check-for-regexp' uses a somewhat non-standard technique of searching for messages: it searches them from the back, from the end of the tprocess's buffer. I don't know what motivated this, but I find it odd. I find one of its callees, tramp-search-regexp, particularly suspicious: (defun tramp-search-regexp (regexp) "Search for REGEXP backwards, starting at point-max."" (goto-char (point-max)) ;; We restrict ourselves to the last 256 characters. There were ;; reports of a shell command "git ls-files -zco --exclude-standard" ;; with 85k files involved, which has blocked Tramp forever. (re-search-backward regexp (max (point-min) (- (point) 256)) 'noerror)) See the comment there? Only 256 characters back are inspected. So, finally, here's my conjecture: 1. Tramp goes into 'tramp-wait-for-regexp'. tprocess's buffer already the message that 'found' is supposed to return, but it also has a lot more stuff, say a lot of JSONRPC data from the LSP server that also came into that tprocess buffer and is awaiting to be delivered to jprocess. 2. This data is for piping into jprocess, where the JSONRPC message will be decoded, but it will probably never arrive at its destination. 3. 'found' will be nil in tramp-wait-for-regexp, because of the tramp-search-regexp limitation. 4. tramp-wait-for-regexp will issue the "risky" accept-process-output call. 5. there is no more data that accept-process-output wants to put in the buffer, because the LSP server is fine for the moment. 6. Emacs hang Just a conjecture. More comments to your comments. > Jsonrpc needs its own markers because N messages can arrive at any > given time and the buffer can not be deleted after each message. Jsonrpc operates in jprocess's buffer, which is separate. > I have a vague feeling, that Tramp could be improved with a work queue > such that requests to tramp from notification or timer threads get > blocked while another tramp command is still waiting for a > reply. There are no (usable) threads in Emacs. Timers are events, and so are runs of each processe's process filter. Those two are what creates asynchronicity and the emulation of simultaneity in Emacs. When jprocess's filter sees a whole JSONRPC message, it calls the message handler. > Joao: "markers in the process buffer is what is used commonly" Markers are positions in the buffer that maintain relative positions if you add text before them. Point is a marker, and so is the process-mark, which marks where process last left input. With that concept in place, it's usually easi(er) to code processing of messages. Jo=C3=A3o