From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: =?UTF-8?Q?Jo=C3=A3o_?= =?UTF-8?Q?T=C3=A1vora?= Newsgroups: gmane.emacs.bugs Subject: bug#60088: 29.0.60; Eglot loses messages from gopls Date: Thu, 15 Dec 2022 11:56:15 +0000 Message-ID: <87pmckg9sw.fsf@gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="22435"; mail-complaints-to="usenet@ciao.gmane.io" To: 60088@debbugs.gnu.org, eliz@gnu.org Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Thu Dec 15 12:55:18 2022 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1p5mpa-0005dA-9a for geb-bug-gnu-emacs@m.gmane-mx.org; Thu, 15 Dec 2022 12:55:18 +0100 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1p5mpM-0007M4-8W; Thu, 15 Dec 2022 06:55:04 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1p5mpL-0007Iu-CO for bug-gnu-emacs@gnu.org; Thu, 15 Dec 2022 06:55:03 -0500 Original-Received: from debbugs.gnu.org ([209.51.188.43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1p5mpL-0006jg-45 for bug-gnu-emacs@gnu.org; Thu, 15 Dec 2022 06:55:03 -0500 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1p5mpK-0002al-Fg for bug-gnu-emacs@gnu.org; Thu, 15 Dec 2022 06:55:02 -0500 X-Loop: help-debbugs@gnu.org Resent-From: =?UTF-8?Q?Jo=C3=A3o_?= =?UTF-8?Q?T=C3=A1vora?= Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Thu, 15 Dec 2022 11:55:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 60088 X-GNU-PR-Package: emacs X-Debbugs-Original-To: bug-gnu-emacs@gnu.org, eliz@gnu.org Original-Received: via spool by submit@debbugs.gnu.org id=B.16711053019952 (code B ref -1); Thu, 15 Dec 2022 11:55:02 +0000 Original-Received: (at submit) by debbugs.gnu.org; 15 Dec 2022 11:55:01 +0000 Original-Received: from localhost ([127.0.0.1]:42869 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1p5mpI-0002aS-Vp for submit@debbugs.gnu.org; Thu, 15 Dec 2022 06:55:01 -0500 Original-Received: from lists.gnu.org ([209.51.188.17]:54668) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1p5mpG-0002aM-V4 for submit@debbugs.gnu.org; Thu, 15 Dec 2022 06:54:59 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1p5mpG-0007Ic-L6 for bug-gnu-emacs@gnu.org; Thu, 15 Dec 2022 06:54:58 -0500 Original-Received: from mail-wr1-x42f.google.com ([2a00:1450:4864:20::42f]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1p5mpC-0006hh-0b; Thu, 15 Dec 2022 06:54:58 -0500 Original-Received: by mail-wr1-x42f.google.com with SMTP id o5so2793345wrm.1; Thu, 15 Dec 2022 03:54:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:from:to:cc:subject:date:message-id:reply-to; bh=v7Sa3LfqHTd6Mcwi1lJHo56h8kUWjY8uHwUR+zcuDN0=; b=AAX188XLwaKJOYdkudo2TMMWt9AE86KUDQqWR+IFuNSlHrVGDyUEwcu0jl95KX1iHE +D8rjwWoQrTGyAr5hrgEB7pmbBYFZiYDyb/sV2pkHLQfneXqrM828BGAA6qADMfNK7i4 h5yweWGkQ8tMnHcCc9Ui+Zzc1dY+zdB8pdXFORZUkPwm9VKqVCIQsHYUE7EOonjQFg/a wPg33hBsxZXkcwD+El0n5+hUp3KxEWnWM6UtKwpKnorOc7o5o5+Ns3EmosGx04LXBbap RKxAsgUtgTImYR4ff9t36Nnw64vDOmutvb3+WT7WpPhBn99JeZaspHsK+Vy4GYGtTeyX d1Aw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=v7Sa3LfqHTd6Mcwi1lJHo56h8kUWjY8uHwUR+zcuDN0=; b=e3Y8QXM7pAYU+gLATRIenqcxoKgwtjyF24gLsfR5r4fG7LdRlWA/uu2AKY1JxvAERe KQyEeuCmT/C2doPrt7Fe0U+kTxkj5C1IiO9M29EmT0bPpHpxLgeJFw6RYQznFexQV2bF 6/dmrYKtLu515biTWSiEHdkb4qK1PO7g1J8+mOY1B2EGa9g7+ZyWneOplORQzFQo0OSm KXMj7ry20FupbuPLInO4Fn3SbVDJt0NqLk1qo0c/IyElh+/ODoksQzegvtAZr2JnyOKM o73owds1/Nc0Uyp3NjqkPcEA2KExatJndhLZ8sktQQUZuoFgtRMyaKOKBbqx88yunS/C OO8A== X-Gm-Message-State: AFqh2kpx9RyAYT6SBI7cE+w/Xiu1Gs+JtfkCFe0c5ALT79768M3SOyXg dW9Rv+POErKeON+mrqTzmE992VjgsOg= X-Google-Smtp-Source: AMrXdXutIb0BiadVNwcqKcmKOn/bp2+TMV0k/wXPBLHLSV4crNRzv/gpFko3HahQ2nCvQ85apLdzYQ== X-Received: by 2002:a5d:58f8:0:b0:256:1d9b:bd4b with SMTP id f24-20020a5d58f8000000b002561d9bbd4bmr4492182wrd.55.1671105291224; Thu, 15 Dec 2022 03:54:51 -0800 (PST) Original-Received: from krug ([87.196.73.63]) by smtp.gmail.com with ESMTPSA id f9-20020a05600c154900b003b49bd61b19sm2483462wmg.15.2022.12.15.03.54.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 15 Dec 2022 03:54:50 -0800 (PST) Received-SPF: pass client-ip=2a00:1450:4864:20::42f; envelope-from=joaotavora@gmail.com; helo=mail-wr1-x42f.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.bugs:251076 Archived-At: [Eli, I have CC'ed you directly because this bug concerns a technique to avoid recursive process filters, which seems like an area you are expert in.] Hi,=20 In some situations, Eglot is losing some messages from the Go language server, 'gopls'. The problem has been reproduced and identified. For context, this started in: * https://github.com/joaotavora/eglot/issues/587#issuecomment-1221072833 * https://github.com/golang/go/issues/54559 (In the former link, there are reports of what seem to be other problems. The reports are missing clear reproduction recipes, so they may not be the same problem I'm about to describe.) Reproduction: Ensure go-mode is installed from https://melpa.org/#/go-mode $ go install golang.org/x/tools/gopls@latest $ git clone git clone https://github.com/google/nftables $ emacs -Q -f package-initialize nftables/nftables_test.go M-x eglot The last step will connect to the server, exchange some messages. It will not block Emacs, but the Eglot session will be completely non-functional. Analysis: Many essential JSONRPC messages have not been exchanged, they have been lost. The bytes of the lost messages from the gopls side can be seen in the process buffer (M-x list-processes, then click the link to the hidden process buffer). The problem happens in jsonrpc.el's jsonrpc--process-filter. The code does not expect this function to be called recursively, but that can happen if the client code invoked by its jsonrpc--connection-receive callee eventually sends a follow-up message to the server. This is a common scenario, especially during LSP initialiation. It has been working fine for a number of years and a large number of servers. However: * if that follow-up message is large enough (this is why the largish nftables_test.go file is needed) AND * the server has already sent us some output with just the right (wrong) timing. then the process-send-string call that is eventually reached will in fact cause more output to arrive from the process and a recursive invocation of the process filter. Resolution: I've looked at a number of ways to solve this problem, from avoiding the follow up message in Eglot's side to different techniques on the jsonrpc side. None of them are as satisfactory as the patch after my sig, which simply re-schedules the would-be recursive call to re-run from within a timer stack as soon as possible. The only problem I can envision with this approach would be if multiple recursive calls are launched from the same jsonrpc--connection-receive stack frame and would somehow not be executed in the correct order. That doesn't seem to be a problem after some experiments like (progn (run-at-time 0 nil #'insert "a") (run-at-time 0 nil #'insert "b") (run-at-time 0 nil #'insert "c")) which inserts the predicatbly sane "abc". If this in-order execution is somehow not guaranteed, there is the alternate more complex technique of inserting the output into the process buffer immediately, and schedule a single processing call thereafter. I will install the following patch soon unless someone objects to this approach. Jo=C3=A3o diff --git a/lisp/jsonrpc.el b/lisp/jsonrpc.el index 90833e1c1d..76bbf28138 100644 --- a/lisp/jsonrpc.el +++ b/lisp/jsonrpc.el @@ -548,11 +548,27 @@ jsonrpc--process-sentinel (delete-process proc) (funcall (jsonrpc--on-shutdown connection) connection))))) =20 -(defun jsonrpc--process-filter (proc string) +(defvar jsonrpc--in-process-filter nil + "Non-nil if inside `jsonrpc--process-filter'.") + +(cl-defun jsonrpc--process-filter (proc string) "Called when new data STRING has arrived for PROC." + (when jsonrpc--in-process-filter + ;; Problematic recursive process filters may happen if + ;; `jsonrpc--connection-receive', called by us, eventually calls + ;; client code which calls `process-send-string' (which see) to, + ;; say send a follow-up message. If that happens to writes enough + ;; bytes for additional output to be received, we have a problem. + ;; + ;; In that case, remove recursiveness by re-scheduling ourselves to + ;; run from within a timer as soon as possible. + + (run-at-time 0 nil #'jsonrpc--process-filter proc string) + (cl-return-from jsonrpc--process-filter)) (when (buffer-live-p (process-buffer proc)) (with-current-buffer (process-buffer proc) (let* ((inhibit-read-only t) + (jsonrpc--in-process-filter t) (connection (process-get proc 'jsonrpc-connection)) (expected-bytes (jsonrpc--expected-bytes connection))) ;; Insert the text, advancing the process marker.