From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: =?UTF-8?Q?Jo=C3=A3o_?= =?UTF-8?Q?T=C3=A1vora?= Newsgroups: gmane.emacs.bugs Subject: bug#67945: 30.0.50; Jsonrpc trouble dealing with nested synchronous requests Date: Wed, 20 Dec 2023 23:30:08 +0000 Message-ID: Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="000000000000d5f06b060cf95da5" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="33673"; mail-complaints-to="usenet@ciao.gmane.io" To: 67945@debbugs.gnu.org, daniel@dpettersson.net Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Thu Dec 21 00:31:22 2023 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1rG625-0008av-NJ for geb-bug-gnu-emacs@m.gmane-mx.org; Thu, 21 Dec 2023 00:31:21 +0100 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rG61m-0006Re-KC; Wed, 20 Dec 2023 18:31:02 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rG61j-0006RQ-LB for bug-gnu-emacs@gnu.org; Wed, 20 Dec 2023 18:30:59 -0500 Original-Received: from debbugs.gnu.org ([2001:470:142:5::43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rG61j-0004i1-C5 for bug-gnu-emacs@gnu.org; Wed, 20 Dec 2023 18:30:59 -0500 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1rG61m-00034R-Ra for bug-gnu-emacs@gnu.org; Wed, 20 Dec 2023 18:31:02 -0500 X-Loop: help-debbugs@gnu.org Resent-From: =?UTF-8?Q?Jo=C3=A3o_?= =?UTF-8?Q?T=C3=A1vora?= Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Wed, 20 Dec 2023 23:31:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 67945 X-GNU-PR-Package: emacs X-Debbugs-Original-To: "simon254--- via Bug reports for GNU Emacs, the Swiss army knife of text editors" , Daniel Pettersson Original-Received: via spool by submit@debbugs.gnu.org id=B.170311505811272 (code B ref -1); Wed, 20 Dec 2023 23:31:02 +0000 Original-Received: (at submit) by debbugs.gnu.org; 20 Dec 2023 23:30:58 +0000 Original-Received: from localhost ([127.0.0.1]:42040 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rG61h-0002uo-46 for submit@debbugs.gnu.org; Wed, 20 Dec 2023 18:30:57 -0500 Original-Received: from lists.gnu.org ([2001:470:142::17]:49708) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1rG61d-0002eS-OB for submit@debbugs.gnu.org; Wed, 20 Dec 2023 18:30:55 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rG61K-0006QN-GT for bug-gnu-emacs@gnu.org; Wed, 20 Dec 2023 18:30:38 -0500 Original-Received: from mail-lj1-x230.google.com ([2a00:1450:4864:20::230]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1rG61F-0004Yc-Nr for bug-gnu-emacs@gnu.org; Wed, 20 Dec 2023 18:30:31 -0500 Original-Received: by mail-lj1-x230.google.com with SMTP id 38308e7fff4ca-2cc794df8aaso1819141fa.0 for ; Wed, 20 Dec 2023 15:30:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1703115021; x=1703719821; darn=gnu.org; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=OAXT6G++/8lQKt6EvKIw1MZdvCMPSI308cvcfUEswa4=; b=TjsrrZP3POxtahdg141xKMp/zDxebL+5li1ArT9a2spSlwLHC5elqEXIiq27lndzsK D+Pqm9BSxPv5VGjrzas3kHMw3KkmSMzi9NJ60IL/jj+2gyQ0UdzXfNsklNXROAfo/2Nh Y2+8hhcnZKp+WUUmAcaS53Bv9AB0nT17JXvWJ00780v/UOTVP0lOAHVr8IYkjYge8Rkf CTgJrgSD7uhxceseTS1tRW/fQHFEKO/R484VPOHopVb65lncit9JxsFy/NdfnryPX/8l 8GZPnFMrcCBy7dpSGlzkkHhNboWBjwmtPMZ6Bn79tkSCJeA3ei115ECuVVo8oVgazRiV fMzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703115021; x=1703719821; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=OAXT6G++/8lQKt6EvKIw1MZdvCMPSI308cvcfUEswa4=; b=MV0dUvcRO8DsAZbiYQx9MZkeUrIMf8pVs16fgniLyg9LL5skDwzp0nDdzAWJCBE5Fw eETelaHl05Ba7PcCfwpCgzMhmYF1SptSiL+kNyf6VEN/AadYjoOUObvvukxU0ToSa2Eh RNQlRx7+rVvRN49n95/AjOaI1WG5G5Gt3Io++SQehgML7IwcYEpl7mf6IWS8Oyaj7DXl 24wIXak+KCncVXeBCWSyAjtOaR2s18/YzfMD3uBHQXtcFaY7YxVOuG/xtzM8HCltcwCF LD+SczBwUmsTPR480ZL9DUQoYSkqcxCNN8Cef5t64dbSlsTf7KueSbgzahyD/a1EgSki +vfQ== X-Gm-Message-State: AOJu0Yx4WbABgR9EBpsfF9ToJ8qbxcMNjwVDusEVHJ1+mLYqtFN+KSVs PUP4++t4zllO/7NYUmyGVRptdULYxvlI+N2gcVWSq+XqqMU= X-Google-Smtp-Source: AGHT+IHTu84EoHSS0RJJhgJq9ljfHc7AqYO0sTjE0Jd21PZtcQTLemaJKR5G0TfakXGVAuCgBFqbKlfJEouoc/3BBKc= X-Received: by 2002:a2e:9643:0:b0:2cc:5cf8:13d with SMTP id z3-20020a2e9643000000b002cc5cf8013dmr3838929ljh.102.1703115020938; Wed, 20 Dec 2023 15:30:20 -0800 (PST) Received-SPF: pass client-ip=2a00:1450:4864:20::230; envelope-from=joaotavora@gmail.com; helo=mail-lj1-x230.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.bugs:276600 Archived-At: --000000000000d5f06b060cf95da5 Content-Type: text/plain; charset="UTF-8" Hi, I've come across a rather complicated bug in jsonrpc.el that affects a certain class of JSON-RPC applications. Eglot, on of the more important `jsonrpc.el` clients (perhaps the most) is not affected, as far as I can tell. I already have a fix for this bug, but I am filing this report as a reference for the commit message and to collect comments. I was made aware of this bug by Daniel Pettersson, who is experimenting with integrating JSONRPC in his DAP client, dape.el. DAP's based transport protocol isn't really JSON-RPC based but jsonrpc.el can be accomodated to support it. However the bug can be triggered with a sufficiently insidious and plain JSON-RPC application. The attached files jbug.el and jbug.py worked on by Daniel and me demonstrate the problem. * jbug.py is a JSON-RPC server that upon receiving the ":ping"" request immediately notifies the client with a ":hello"" notification. Then it waits briefly 0.1 seconds before reponding to the ":ping" request with ":pong". * jbug.el is a JSON-RPC client that sends a ":ping" request and waits for a response. While waiting for that response, it receives the ":hello" notification and issues a synchronous second ":slow" request inside the notification handler. Both requests are synchronous and performed with the Elisp function 'jsonrpc-request' As the name implies, the server takes longer to answer the ":slow" request (about 0.5 seconds) Before the patch that I'm about to push, the response to the ":slow" request is never seen by the client. The notification handler, which runs in a timer, simply exists non-locally. The reasons for this are somewhat complicated, but now well understood. Briefly, the synchronicity of 'jsonrpc-request' is achieved by having 'accept-process-output' wrapped in a try-catch block. A continuation handler running in a timer will eventually throw to the catch tag, so the next form after 'jsonrpc-request' can be executed. During the wait, JSON-RPC notifications may come in and they often do, but in most JSON-RPC applications (Eglot, at least), the notification handlers don't issue what will become a nested 'jsonrpc-request'. And even if they were to do that, they would only run into this problem if that second inner request ends at a later time than the first request. In dape.el and in the jbug.el file attached, this can happen. The result will be that the outer catch tag of the first outer request is "thrown to" when the response arrives, but that also unwinds the stack of the second inner request, which leads to the application never seeing the response that will appear only later. In the fix I'm about to push, the solution is to record the nesting in an Elisp structure and prevent throwing to the outer request when its response arrives. Rather, we record that desired continuation to run only when the inner request has resolved fully. The throwing will happen only when we are back at the first outer requests's 'accept-process-output'. This makes both responses be seen by the JSON-RPC application, albeit in reverse order. I think this isn't a problem in real life, because from the jsonrpc.el programmer's perspective a blocking jsonrpc-request call is only meant to preserve the order of execution of the forms in which that call. It's _not_ the relative order of multiple "concurrent" 'jsonrpc-request' calls, because this concurrency isn't really defined in the single-threaded Elisp programming model. I attach the two files jbug.el and jbug.py. --000000000000d5f06b060cf95da5 Content-Type: application/octet-stream; name="jbug.py" Content-Disposition: attachment; filename="jbug.py" Content-Transfer-Encoding: base64 Content-ID: X-Attachment-Id: f_lqeejv7w1 IyEvdXNyL2Jpbi9lbnYgcHl0aG9uMwojIGpidWcucHkKaW1wb3J0IHN5cwppbXBvcnQganNvbgpp bXBvcnQgYXN5bmNpbwpmcm9tIGpzb25ycGNzZXJ2ZXIgaW1wb3J0IG1ldGhvZCwgYXN5bmNfZGlz cGF0Y2gsIFN1Y2Nlc3MsIEVycm9yCgpAbWV0aG9kCmFzeW5jIGRlZiBwaW5nKCk6CiAgICBzZW5k X2pzb24oeyJtZXRob2QiOiAiaGVsbG8iLCAicGFyYW1zIjoge319KQogICAgYXdhaXQgYXN5bmNp by5zbGVlcCgwLjEpICAjIFNpbXVsYXRlIGFzeW5jaHJvbm91cyB3b3JrCiAgICByZXR1cm4gU3Vj Y2VzcygicG9uZyIpCgpAbWV0aG9kCmFzeW5jIGRlZiBzbG93KHNlbmRlcik6CiAgICBhd2FpdCBh c3luY2lvLnNsZWVwKDAuNSkKICAgIHJldHVybiBTdWNjZXNzKGYiUmVzcG9uc2Ugd2l0aGluIHtz ZW5kZXJ9IikKCmRlZiBzZW5kX2pzb24oZGF0YSk6CiAgICBkYXRhID0gZGljdChkYXRhKQogICAg ZGF0YVsianNvbnJwYyJdID0gIjIuMCIKICAgIGpzb25fZGF0YSA9IGpzb24uZHVtcHMoZGF0YSkK ICAgIHdyaXRlX3Jlc3BvbnNlKGpzb25fZGF0YSkKCmRlZiB3cml0ZV9yZXNwb25zZShyZXNwb25z ZV9zdHIpOgogICAgc3lzLnN0ZG91dC53cml0ZShmIkNvbnRlbnQtbGVuZ3RoOiB7bGVuKHJlc3Bv bnNlX3N0cil9XHJcblxyXG57cmVzcG9uc2Vfc3RyfSIpCiAgICBzeXMuc3Rkb3V0LmZsdXNoKCkK CmFzeW5jIGRlZiBoYW5kbGVfcmVxdWVzdChyZXF1ZXN0X3N0cik6CiAgICB0cnk6CiAgICAgICAg cmVzcG9uc2Vfc3RyID0gYXdhaXQgYXN5bmNfZGlzcGF0Y2gocmVxdWVzdF9zdHIpCiAgICAgICAg d3JpdGVfcmVzcG9uc2UocmVzcG9uc2Vfc3RyKQogICAgZXhjZXB0IEV4Y2VwdGlvbiBhcyBlOgog ICAgICAgIGVycm9yX3Jlc3BvbnNlID0gRXJyb3Ioc3RyKGUpKQogICAgICAgIHdyaXRlX3Jlc3Bv bnNlKHN0cihlcnJvcl9yZXNwb25zZSkpCgpkZWYgcmVhZF9tZXNzYWdlKCk6CiAgICBjb250ZW50 X2xlbmd0aCA9IHN5cy5zdGRpbi5yZWFkbGluZSgpLnN0cmlwKCkKICAgIHN5cy5zdGRpbi5yZWFk bGluZSgpCiAgICBtZXNzYWdlX2xlbmd0aCA9IGludChjb250ZW50X2xlbmd0aC5zcGxpdCgiOiIp WzFdLnN0cmlwKCkpCiAgICBtZXNzYWdlID0gc3lzLnN0ZGluLnJlYWQobWVzc2FnZV9sZW5ndGgp CiAgICByZXR1cm4gbWVzc2FnZQoKYXN5bmMgZGVmIG1haW4oKToKICAgIGxvb3AgPSBhc3luY2lv LmdldF9ldmVudF9sb29wKCkKCiAgICB3aGlsZSBUcnVlOgogICAgICAgIHRyeToKICAgICAgICAg ICAgbWVzc2FnZSA9IGF3YWl0IGxvb3AucnVuX2luX2V4ZWN1dG9yKE5vbmUsIHJlYWRfbWVzc2Fn ZSkKICAgICAgICAgICAgbG9vcC5jcmVhdGVfdGFzayhoYW5kbGVfcmVxdWVzdChtZXNzYWdlKSkK ICAgICAgICBleGNlcHQgRU9GRXJyb3I6CiAgICAgICAgICAgIGJyZWFrCgppZiBfX25hbWVfXyA9 PSAiX19tYWluX18iOgogICAgYXN5bmNpby5ydW4obWFpbigpKQo= --000000000000d5f06b060cf95da5 Content-Type: application/octet-stream; name="jbug.el" Content-Disposition: attachment; filename="jbug.el" Content-Transfer-Encoding: base64 Content-ID: X-Attachment-Id: f_lqeejv7o0 Ozs7IGpidWcuZWwgLS0tIGFzZCAgICAgICAgICAgICAgICAgICAgICAgICAgIC0qLSBsZXhpY2Fs LWJpbmRpbmc6IHQ7IC0qLQoocmVxdWlyZSAnanNvbnJwYykKCihkZWZjbGFzcyBqYnVnLXRlc3Qt Y29ubmVjdGlvbiAoanNvbnJwYy1wcm9jZXNzLWNvbm5lY3Rpb24pICgodGhpbmdzIDppbml0Zm9y bSBuaWwpKSkKCihjbC1kZWZtZXRob2QgamJ1Zy1oYW5kbGUtbm90aWZpY2F0aW9uIChjb25uICht IChlcWwgaGVsbG8pKSBfKQogIChwdXNoIChqc29ucnBjLXJlcXVlc3QgY29ubiA6c2xvdyAobGlz dCA6c2VuZGVyIChzeW1ib2wtbmFtZSBtKSApKQogICAgICAgIChzbG90LXZhbHVlIGNvbm4gJ3Ro aW5ncykpKQoKKGRlZnZhciBqYnVnLXRlc3QtY2xpZW50IG5pbCkKCgooZGVmdW4gamJ1Zy1ldmVu dHMgKCkKICAoaW50ZXJhY3RpdmUpCiAgKHdpdGgtY3VycmVudC1idWZmZXIKICAgICAgKGpzb25y cGMtZXZlbnRzLWJ1ZmZlciBqYnVnLXRlc3QtY2xpZW50KQogICAgKGxldCAoKGluaGliaXQtcmVh ZC1vbmx5IHQpKQogICAgICAoZXJhc2UtYnVmZmVyKSkKICAgIChkaXNwbGF5LWJ1ZmZlciAoY3Vy cmVudC1idWZmZXIpKSkpCgooZGVmdW4gamJ1ZyAoKQogIChpbnRlcmFjdGl2ZSkKICAod2hlbiBq YnVnLXRlc3QtY2xpZW50CiAgICAoanNvbnJwYy1zaHV0ZG93biBqYnVnLXRlc3QtY2xpZW50KQog ICAgKGpidWctZXZlbnRzKSkKICAoc2V0cSBqYnVnLXRlc3QtY2xpZW50CiAgICAgICAgKG1ha2Ut aW5zdGFuY2UgJ2pidWctdGVzdC1jb25uZWN0aW9uCiAgICAgICAgICAgICAgICAgICAgICAgOm5h bWUgInRlc3QgY2xpZW50IgogICAgICAgICAgICAgICAgICAgICAgIDpyZXF1ZXN0LWRpc3BhdGNo ZXIgJ2lnbm9yZQogICAgICAgICAgICAgICAgICAgICAgIDpub3RpZmljYXRpb24tZGlzcGF0Y2hl ciAjJ2pidWctaGFuZGxlLW5vdGlmaWNhdGlvbgogICAgICAgICAgICAgICAgICAgICAgIDpwcm9j ZXNzIChsYW1iZGEgKF9jb25uKQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgKG1h a2UtcHJvY2VzcwogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDpuYW1lICJkdW1t eSBzZXJ2ZXIiCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgOnN0ZGVyciAoZ2V0 LWJ1ZmZlci1jcmVhdGUKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAoZm9ybWF0ICIqdGVzdCBjbGllbnQgc3RkZXJyKiIpKQogICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgIDpjb21tYW5kICcoInB5dGhvbjMiICJqYnVnLnB5IikKICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICA6Y29ubmVjdGlvbi10eXBlICdwaXBlCiAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgOmNvZGluZyAndXRmLTgtZW1hY3MtdW5peAogICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgIDpub3F1ZXJ5IHQpKSkpKQoKKHByb2duCiAgKGpi dWcpCiAgKGpidWctZXZlbnRzKSA7IGZvciBkZWJ1Z2dpbmcKICAocHVzaCAoanNvbnJwYy1yZXF1 ZXN0IGpidWctdGVzdC1jbGllbnQgOnBpbmcgKG1ha2UtaGFzaC10YWJsZSkpCiAgICAgICAgKHNs b3QtdmFsdWUgamJ1Zy10ZXN0LWNsaWVudCAndGhpbmdzKSkKICAoc2F2ZS1leGN1cnNpb24KICAg IChpbnNlcnQgKGZvcm1hdCAiXG47OyAlUyIgKHNsb3QtdmFsdWUgamJ1Zy10ZXN0LWNsaWVudCAn dGhpbmdzKSkpKSkKOzsgKCJwb25nIiAiUmVzcG9uc2Ugd2l0aGluIGhlbGxvIikKOzsgKCJwb25n IiAiUmVzcG9uc2Ugd2l0aGluIGhlbGxvIikKCjs7ICgicG9uZyIgIlJlc3BvbnNlIHdpdGhpbiBz ZWNvbmRFdmVudCIpCjs7ICgicG9uZyIgIlJlc3BvbnNlIHdpdGhpbiBzZWNvbmRFdmVudCIpCjs7 ICgicG9uZyIgIlJlc3BvbnNlIHdpdGhpbiBzZWNvbmRFdmVudCIpCjs7ICgiUmVzcG9uc2Ugd2l0 aGluIHNlY29uZEV2ZW50IiAicG9uZyIpCjs7ICgiUmVzcG9uc2Ugd2l0aGluIHNlY29uZEV2ZW50 IiAicG9uZyIpCgo= --000000000000d5f06b060cf95da5--