From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Eli Zaretskii Newsgroups: gmane.emacs.devel Subject: Re: call-process should not block process filters from running Date: Sun, 02 Jul 2023 08:45:42 +0300 Message-ID: <837criq321.fsf@gnu.org> References: <83cz1fvjef.fsf@gnu.org> <83h6qnpieb.fsf@gnu.org> Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="14038"; mail-complaints-to="usenet@ciao.gmane.io" Cc: emacs-devel@gnu.org To: Spencer Baugh Original-X-From: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Sun Jul 02 07:46:10 2023 Return-path: Envelope-to: ged-emacs-devel@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1qFpuR-0003Mo-KE for ged-emacs-devel@m.gmane-mx.org; Sun, 02 Jul 2023 07:46:07 +0200 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qFpta-0000xh-HP; Sun, 02 Jul 2023 01:45:14 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qFptV-0000xL-WB for emacs-devel@gnu.org; Sun, 02 Jul 2023 01:45:11 -0400 Original-Received: from fencepost.gnu.org ([2001:470:142:3::e]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qFptV-0008AA-Gf; Sun, 02 Jul 2023 01:45:09 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org; s=fencepost-gnu-org; h=References:Subject:In-Reply-To:To:From:Date: mime-version; bh=iXFQqX0VAeF6egKL1sqqHrTP4hbeNQhC1ZYb7DJYhk4=; b=ZcGpVa4vONcT cCToJ0O/OlngxqrA4TziGJtA8Ep+gBpwMY51uZRPFUAf9rgQGE/ex/oLvLbCo0ozTQ0ihi83aDo44 o844EA5SRk796XaYf7JYpc8qwXFCzfOcEoD2TA9/Fl9DSoOGUv+4I16n+KkvfWdCYsfOFmkefApYK 1jWIzO0fkarpHIRsmvqtp2T97nzn6TPPrfNirS8WIBnxMvpbsv0IrdLXVHbp0XivSSYK1cqbHJfxL pHvtlwvB9WNXdpEiaMgcD21vHsL1xGQ7ATaKBnuIetboMm8GeJtypifNF82ZLYtPDObOSEaai5MKh lcddG0YeFJ2xb9ZWyNcxvA==; Original-Received: from [87.69.77.57] (helo=home-c4e4a596f7) by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qFptU-0004Gb-VE; Sun, 02 Jul 2023 01:45:09 -0400 In-Reply-To: (message from Spencer Baugh on Sat, 01 Jul 2023 15:17:39 -0400) X-BeenThere: emacs-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Emacs development discussions." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Original-Sender: emacs-devel-bounces+ged-emacs-devel=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.emacs.devel:307345 Archived-At: > From: Spencer Baugh > Date: Sat, 01 Jul 2023 15:17:39 -0400 > > > We read from pipe in chunks, and my idea was to leave the reading code > > intact, just call wait_reading_process_output each iteration through > > the reading loop. Long-running call-process calls spend their time in > > this loop, reading the process output that takes long time to produce. > > So that's where I envision most of the gains will come from. > > That's certainly better than nothing, but that won't run other Lisp > while the call-process process is not producing output. Just as one > degenerate example, call-process on "sleep 30" won't run any Lisp. More > realistically, expensive computations which take a while to produce > output ("factor $BIGNUM" for example) will not run much Lisp. From > looking at call-process usage, I think most are of that form. So I > don't think this will solve the problem. My suggestion is to try first and only reject this idea if it indeed is not useful in practice. The "sleep 30" example is not interesting, IMO: Emacs has sleep-for which will do the same while still reading from subprocesses and letting timers run, so if a Lisp program calls "sleep 30" it probably actually wants Emacs to do nothing during that time. So IMO we should find interesting practical use cases which run long-running programs via call-process, and see what this idea gives us in those cases, before deciding whether it makes sense. To see whether the idea "holds water", it is not necessary to implement it in its entirety: it is enough to count the number of iterations through the loop that reads the output (perhaps also examining the effect of making CALLPROC_BUFFER_SIZE_MIN smaller), because this will tell us how many opportunities for calling wait_reading_process_output we will have. > > As for waitpid, I'd initially leave that alone, and only add the > > wait_reading_process_output to the loop that protects that from > > signals. IME, by the time we get to waitpid after reading the last > > portion of the process output, the process have either already exited > > or will do so almost instantly. If we discover this is not so, we > > could later replace waitpid with some wait with timeout in a loop. > > I see. But if that's the case, then that's not really any better than > just calling reading input and running other Lisp after that loop is > done, which of course we already do. Because, if waitpid returns > immediately anyway, it doesn't really matter if we run Lisp before it or > run Lisp after it. That's what I'm saying, yes. But if we find important cases where a program sends EOF to its stdout and then proceeds with some lengthy processing (which then forces us to wait in waitpid for a long time), we could emulate waitpid using different APIs that wait with timeouts. > > I didn't mean to throw away the reading loop and waitpid, because > > doing so will require to write a lot of non-trivial C code, and goes > > against what I consider to be the main advantage of my idea. > > Yes, I understand that you don't want to introduce bugs into > call-process. IMO that's why making a completely new function instead > of modifying call-process is a better bet. It might be a better bet from the POV of not breaking call-process, but it still runs the risk that the new API will have bugs which call-process doesn't. If we want the new API to be attractive, this risk is still important and real. > But if you would prefer a modification to call-process I'm quite happy > to write non-trivial C code, as long as the problem actually gets > solved. I prefer that we try the minimal change I described, and only consider more complicated solutions if that idea turns out to be not useful enough.