From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Eli Zaretskii Newsgroups: gmane.emacs.devel Subject: Re: call-process should not block process filters from running Date: Sat, 01 Jul 2023 21:59:40 +0300 Message-ID: <83h6qnpieb.fsf@gnu.org> References: <83cz1fvjef.fsf@gnu.org> Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="15395"; mail-complaints-to="usenet@ciao.gmane.io" Cc: emacs-devel@gnu.org To: Spencer Baugh Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Sat Jul 01 20:59:33 2023 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1qFfoi-0003qS-Qm for ged-emacs-devel@m.gmane-mx.org; Sat, 01 Jul 2023 20:59:33 +0200 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qFfoM-0007ER-Hc; Sat, 01 Jul 2023 14:59:10 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qFfoL-0007E0-0z for emacs-devel@gnu.org; Sat, 01 Jul 2023 14:59:09 -0400 Original-Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qFfoK-0007KA-62; Sat, 01 Jul 2023 14:59:08 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date: mime-version; bh=cymHVZWCZpcri4x3E4CrFlJMmhs7BU++a4edeRWhqBI=; b=Ane5MxyksKvH YXwAVQ8tv6NK8VRzNPQkhmF43dodCHwW0pqQu/G3meBDSKBa0E0MQkz4ZnVHoLtmrQxwBcOYv+uMU kdHNlbvvJ0Q+18kkJw2UgMyvXzl7nW8zXgAcZ1+T+Zm1lBEVfnqZhs2reznm0rva2Y+uAxoe5x39D JswazUiorauMqW35maThin2WpeokLw+vDCvV2tAXVykNtX2UbdCmysn10KOGXhxTz2BuTs4Xh6aVG wQh6Gx+X/VYnxuXK7D3gMwVm5Mj7PWNm9cVGP1wBp99VQ0Wa3+3Dzgq1iCWHt+CToL841HFFOSWXI hzbtL1JZdV1EyBPVxS2BIA==; Original-Received: from [87.69.77.57] (helo=home-c4e4a596f7) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qFfoJ-0002B7-Ed; Sat, 01 Jul 2023 14:59:07 -0400 In-Reply-To: (message from Spencer Baugh on Sat, 01 Jul 2023 14:24:59 -0400) X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.devel:307335 Archived-At: > From: Spencer Baugh > Date: Sat, 01 Jul 2023 14:24:59 -0400 > > >> And AFAIU, such a way does exist. The implementation of call-process > >> actually forks a subprocess, then reads from its pipe until EOF, and > >> then waits in waitpid for the subprocess to exit. Both the reading > >> and the waiting are done in a loop. So one way of making call-process > >> less blocking is by adding to these loops calls to > >> wait_reading_process_output, like we do, for example, in sleep-for, > >> but conditioned by some new variable exposed to Lisp. Lisp programs > >> which want this behavior could then activate it by binding that new > >> variable. This could give us most or all of the advantages of > >> non-blocking API without most disadvantages. (We'd still need to move > >> to this functionality on a case by case basis, because some of the > >> considerations against that could still be valid in individual cases.) > > > > This sounds great, I would be happy to implement this. I think we > > would also want a tiny wrapper in Lisp which binds this new variable > > then calls call-process, rather than having lots of programs binding > > the variable directly, to make it easier to change the implementation > > strategy in the future. > > > > I'll work on implementing this new variable. If you have any other > > suggestions for it, let me know. > > Just to sanity check before I go down the wrong path: When this variable > is set, instead of doing the reading from the subprocess's pipe in > call-process, we'll need to do it in wait_reading_process_output, so > that other Lisp can run. We can't just add in calls to > wait_reading_process_output alongside the existing calls to read, > because read is blocking, and we need to process other input if there is > some, even if there's no input from the process under call_process. A > similar change will need to happen for waitpid handling. We read from pipe in chunks, and my idea was to leave the reading code intact, just call wait_reading_process_output each iteration through the reading loop. Long-running call-process calls spend their time in this loop, reading the process output that takes long time to produce. So that's where I envision most of the gains will come from. As for waitpid, I'd initially leave that alone, and only add the wait_reading_process_output to the loop that protects that from signals. IME, by the time we get to waitpid after reading the last portion of the process output, the process have either already exited or will do so almost instantly. If we discover this is not so, we could later replace waitpid with some wait with timeout in a loop. I didn't mean to throw away the reading loop and waitpid, because doing so will require to write a lot of non-trivial C code, and goes against what I consider to be the main advantage of my idea.