From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Spencer Baugh Newsgroups: gmane.emacs.devel Subject: Re: call-process should not block process filters from running Date: Sat, 01 Jul 2023 15:17:39 -0400 Message-ID: References: <83cz1fvjef.fsf@gnu.org> <83h6qnpieb.fsf@gnu.org> Mime-Version: 1.0 Content-Type: text/plain Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="33055"; mail-complaints-to="usenet@ciao.gmane.io" User-Agent: Gnus/5.13 (Gnus v5.13) To: emacs-devel@gnu.org Cancel-Lock: sha1:oAiScEPmPG8q0gLpLux+ezWd/ZI= Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Sun Jul 02 06:45:33 2023 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1qFoxp-0008Le-BV for ged-emacs-devel@m.gmane-mx.org; Sun, 02 Jul 2023 06:45:33 +0200 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qFox9-00052s-QC; Sun, 02 Jul 2023 00:44:53 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qFg6S-0001Zn-Aq for emacs-devel@gnu.org; Sat, 01 Jul 2023 15:17:52 -0400 Original-Received: from ciao.gmane.io ([116.202.254.214]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qFg6Q-0004AZ-JJ for emacs-devel@gnu.org; Sat, 01 Jul 2023 15:17:51 -0400 Original-Received: from list by ciao.gmane.io with local (Exim 4.92) (envelope-from ) id 1qFg6N-0006XE-UE for emacs-devel@gnu.org; Sat, 01 Jul 2023 21:17:47 +0200 X-Injected-Via-Gmane: http://gmane.org/ Received-SPF: pass client-ip=116.202.254.214; envelope-from=ged-emacs-devel@m.gmane-mx.org; helo=ciao.gmane.io X-Spam_score_int: -16 X-Spam_score: -1.7 X-Spam_bar: - X-Spam_report: (-1.7 / 5.0 requ) BAYES_00=-1.9, HEADER_FROM_DIFFERENT_DOMAINS=0.25, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-Mailman-Approved-At: Sun, 02 Jul 2023 00:44:49 -0400 X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.devel:307340 Archived-At: Eli Zaretskii writes: >> From: Spencer Baugh >> Date: Sat, 01 Jul 2023 14:24:59 -0400 >> >> >> And AFAIU, such a way does exist. The implementation of call-process >> >> actually forks a subprocess, then reads from its pipe until EOF, and >> >> then waits in waitpid for the subprocess to exit. Both the reading >> >> and the waiting are done in a loop. So one way of making call-process >> >> less blocking is by adding to these loops calls to >> >> wait_reading_process_output, like we do, for example, in sleep-for, >> >> but conditioned by some new variable exposed to Lisp. Lisp programs >> >> which want this behavior could then activate it by binding that new >> >> variable. This could give us most or all of the advantages of >> >> non-blocking API without most disadvantages. (We'd still need to move >> >> to this functionality on a case by case basis, because some of the >> >> considerations against that could still be valid in individual cases.) >> > >> > This sounds great, I would be happy to implement this. I think we >> > would also want a tiny wrapper in Lisp which binds this new variable >> > then calls call-process, rather than having lots of programs binding >> > the variable directly, to make it easier to change the implementation >> > strategy in the future. >> > >> > I'll work on implementing this new variable. If you have any other >> > suggestions for it, let me know. >> >> Just to sanity check before I go down the wrong path: When this variable >> is set, instead of doing the reading from the subprocess's pipe in >> call-process, we'll need to do it in wait_reading_process_output, so >> that other Lisp can run. We can't just add in calls to >> wait_reading_process_output alongside the existing calls to read, >> because read is blocking, and we need to process other input if there is >> some, even if there's no input from the process under call_process. A >> similar change will need to happen for waitpid handling. > > We read from pipe in chunks, and my idea was to leave the reading code > intact, just call wait_reading_process_output each iteration through > the reading loop. Long-running call-process calls spend their time in > this loop, reading the process output that takes long time to produce. > So that's where I envision most of the gains will come from. That's certainly better than nothing, but that won't run other Lisp while the call-process process is not producing output. Just as one degenerate example, call-process on "sleep 30" won't run any Lisp. More realistically, expensive computations which take a while to produce output ("factor $BIGNUM" for example) will not run much Lisp. From looking at call-process usage, I think most are of that form. So I don't think this will solve the problem. > As for waitpid, I'd initially leave that alone, and only add the > wait_reading_process_output to the loop that protects that from > signals. IME, by the time we get to waitpid after reading the last > portion of the process output, the process have either already exited > or will do so almost instantly. If we discover this is not so, we > could later replace waitpid with some wait with timeout in a loop. I see. But if that's the case, then that's not really any better than just calling reading input and running other Lisp after that loop is done, which of course we already do. Because, if waitpid returns immediately anyway, it doesn't really matter if we run Lisp before it or run Lisp after it. > I didn't mean to throw away the reading loop and waitpid, because > doing so will require to write a lot of non-trivial C code, and goes > against what I consider to be the main advantage of my idea. Yes, I understand that you don't want to introduce bugs into call-process. IMO that's why making a completely new function instead of modifying call-process is a better bet. But if you would prefer a modification to call-process I'm quite happy to write non-trivial C code, as long as the problem actually gets solved.