From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: =?utf-8?Q?F=C3=A9lix_Baylac_Jacqu=C3=A9?= Newsgroups: gmane.emacs.help Subject: Async process sentinel running exclusively in main thread? Date: Mon, 20 Jun 2022 10:43:52 +0200 Message-ID: <8735fzbu7r.fsf@alternativebit.fr> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="24672"; mail-complaints-to="usenet@ciao.gmane.io" To: help-gnu-emacs@gnu.org Original-X-From: help-gnu-emacs-bounces+geh-help-gnu-emacs=m.gmane-mx.org@gnu.org Mon Jun 20 10:45:53 2022 Return-path: Envelope-to: geh-help-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1o3D2e-0006Cm-2c for geh-help-gnu-emacs@m.gmane-mx.org; Mon, 20 Jun 2022 10:45:52 +0200 Original-Received: from localhost ([::1]:33468 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1o3D2c-0004t9-CT for geh-help-gnu-emacs@m.gmane-mx.org; Mon, 20 Jun 2022 04:45:50 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:43248) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1o3D0s-0004sl-4s for help-gnu-emacs@gnu.org; Mon, 20 Jun 2022 04:44:02 -0400 Original-Received: from relay4-d.mail.gandi.net ([2001:4b98:dc4:8::224]:55205) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1o3D0p-0006MN-Gn for help-gnu-emacs@gnu.org; Mon, 20 Jun 2022 04:44:01 -0400 Original-Received: (Authenticated sender: felix@alternativebit.fr) by mail.gandi.net (Postfix) with ESMTPSA id 8481FE0016 for ; Mon, 20 Jun 2022 08:43:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=alternativebit.fr; s=gm1; t=1655714633; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=J/se1BHsLpzJzvI0Y1LTtUUEWIG43YNLAYIV6uSbyOw=; b=jR21IjuGNfPCKJNVy6caZEfmVqP2RZEROj4Cw15ixbQZrT3OOSH0NBLXGgRed7znp4aM6b Prt9puDny289hAAw2rbjVuQI5sfYBYHA+NDe48UDhY1ed5XIVbSe+hTGZe92tD4UR4lBr9 5f7r98/P0P1216BcAt4d2+3Gxu+L/Ap4KSb1Bmao34wOLjR4wHGxSKTgBBfupI1SywRT3B ZRj/RHb1ne2PnozFu7WdT/s/04BaiZWNrO6KRppHAbgjQ3fJLP7dIuzuy/xYKHbeOltJ48 feUmfJQGPNA/7UF8QzaJ5SADiBiUdzBP+JvrUQ6OvJGISnyzUGs0jeH2MMvx8g== Received-SPF: pass client-ip=2001:4b98:dc4:8::224; envelope-from=felix@alternativebit.fr; helo=relay4-d.mail.gandi.net X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: help-gnu-emacs@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Users list for the GNU Emacs text editor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: help-gnu-emacs-bounces+geh-help-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: "help-gnu-emacs" Xref: news.gmane.io gmane.emacs.help:137936 Archived-At: Hi folks, Let's start with a bit of context: I'd like to test a function spinning up an asynchronous process call using ERT. Sadly, at the moment (Emacs 28.1), there's no way for ERT to cope with asynchronous calls out of the box. You can find a clever trick allowing you to do that online though: Rejeep's ert-async.el library. This trick is based on abusing the accept-process-output function. This library has been written in 2014: at the time, condition-variables were not a thing and accept-process-output was the best option you had to block the main thread and wait for another thread to be done. Things changed since then, we now have better tools to manage concurrency in Emacs! I came to think we could probably re-write this library using condition variables. The overall idea is to wait for the subprocess to be done by blocking the main thread (the one running the ERT BODY test case) using a condition variable. The subprocess would then unblock the main thread using its sentinel. Overall, it transforms the async subprocess call into a synchronous for the test case. Which bring us to the issue: this approach results in deadlocking the main thread. /!\ WARNING: RUNNING THIS SNIPPET WILL DEADLOCK YOUR EMACS INSTANCE!!!! --8<---------------cut here---------------start------------->8--- (defun test-sentinel (process event) (with-mutex mutex (progn (message "Sentinel: running") (setq exit-code 1) (condition-notify cond-var) (message "Sentinel: notified")))) (defun test-make-process () (let* ((run-async-process (lambda () (progn (make-process :name "dummy-async-subprocess" :buffer "ls-buf" :command '("ls") :sentinel 'test-sentinel)))) (run-wait-process (lambda () (progn (with-mutex mutex (while (not exit-code) (message "Wait: Waiting for exit-code") (condition-wait cond-var) (message "Wait: notified"))))))) (progn (setq exit-code nil) (setq mutex (make-mutex "mutex-test")) (setq cond-var (make-condition-variable mutex "cond-test")) (make-thread run-async-process) (funcall run-wait-process)))) (test-make-process) --8<---------------cut here---------------end--------------->8--- This deadlock is quite surprising for an Emacs newcomer like me: the sentinel end up never being executed. According to the make-process documentation, the process is supposed to be attached to the thread it originates from. In this case the run-async-process thread. The condition-wait originating from the main thread shouldn't lock the subprocess sentinel. This issue confused me for quite a while. Until I noticed something quite surprising with the following snippet: Note: this one is safe to run :) --8<---------------cut here---------------start------------->8--- (defun test-sentinel (process event) (with-mutex mutex (progn (message "Sentinel: running") (message (format "Sentinel: run in main thread? %s" (equal (current-t= hread) main-thread))) (setq exit-code 1) (condition-notify cond-var) (message "Sentinel: notified")))) (defun test-make-process () (let* ((run-async-process (lambda () (progn (message (format "run-async-process: run in main thread? %s" = (equal (current-thread) main-thread))) (make-process :name "dummy-async-subprocess" :buffer "ls-buf" :command '("ls") :sentinel 'test-sentinel)))) (run-wait-process (lambda () (progn (with-mutex mutex (while (not exit-code) (message (format "Wait: run in main thread? %s" (equal (c= urrent-thread) main-thread))) (message "Wait: Waiting for exit-code") (condition-wait cond-var) (message "Wait: notified"))))))) (progn (setq exit-code nil) (setq mutex (make-mutex "mutex-test")) (setq cond-var (make-condition-variable mutex "cond-test")) (make-thread run-async-process) (make-thread run-wait-process)))) (test-make-process) --8<---------------cut here---------------end--------------->8--- Message output on Emacs 28.1: --8<---------------cut here---------------start------------->8--- Wait: run in main thread? nil Wait: Waiting for exit-code run-async-process: run in main thread? nil Sentinel: running Sentinel: run in main thread? t Sentinel: notified Wait: notified --8<---------------cut here---------------end--------------->8--- As you can see, while the run-async-process and the run-wait-process functions are each living in their own threads, but the sentinel ends up running in the main thread instead of the run-wait-process one!? Surprising. That explains the deadlock origin though. I couldn't find any mention of the fact that the sentinel function will exclusively run in the main thread in the documentation. Regardless of the thread in which the process is actually executed. Which makes me wonder: Is this a bug or is this an expected behavior? In case it is an expected behavior, is there any way to wait for a subprocess to be done in the main thread using a condition variable without deadlocking Emacs? Thanks a lot for taking the time to read this long message :) F=C3=A9lix