Re: call-process should not block process filters from running

[sbaugh@catern.com]

Eli Zaretskii <eliz@gnu.org> writes:
>> From: Spencer Baugh <sbaugh@janestreet.com>
>> Date: Sat, 01 Jul 2023 15:17:39 -0400
>> 
>> > We read from pipe in chunks, and my idea was to leave the reading code
>> > intact, just call wait_reading_process_output each iteration through
>> > the reading loop.  Long-running call-process calls spend their time in
>> > this loop, reading the process output that takes long time to produce.
>> > So that's where I envision most of the gains will come from.
>> 
>> That's certainly better than nothing, but that won't run other Lisp
>> while the call-process process is not producing output.  Just as one
>> degenerate example, call-process on "sleep 30" won't run any Lisp.  More
>> realistically, expensive computations which take a while to produce
>> output ("factor $BIGNUM" for example) will not run much Lisp.  From
>> looking at call-process usage, I think most are of that form.  So I
>> don't think this will solve the problem.
>
> My suggestion is to try first and only reject this idea if it indeed
> is not useful in practice.
>
> The "sleep 30" example is not interesting, IMO: Emacs has sleep-for
> which will do the same while still reading from subprocesses and
> letting timers run, so if a Lisp program calls "sleep 30" it probably
> actually wants Emacs to do nothing during that time.
>
> So IMO we should find interesting practical use cases which run
> long-running programs via call-process, and see what this idea gives
> us in those cases, before deciding whether it makes sense.  To see
> whether the idea "holds water", it is not necessary to implement it in
> its entirety: it is enough to count the number of iterations through
> the loop that reads the output (perhaps also examining the effect of
> making CALLPROC_BUFFER_SIZE_MIN smaller), because this will tell us
> how many opportunities for calling wait_reading_process_output we will
> have.

Alright, with this trivial patch:

diff --git a/src/callproc.c b/src/callproc.c
index 6f3d4fad9be..1308228ab6c 100644
--- a/src/callproc.c
+++ b/src/callproc.c
@@ -778,6 +778,8 @@ call_process (ptrdiff_t nargs, Lisp_Object *args, int filefd,
 	      nread += this_read;
 	      total_read += this_read;
 
+	      wait_reading_process_output(-1, -1, 0, display_on_the_fly,
+					  NULL, NULL, 0);
 	      if (display_on_the_fly)
 		break;
 	    }

the following code (when run in the Emacs source tree) evaluates to 3
instead of 0 for me:

(length (let (marks
              (timer (run-at-time "0 sec" .1
                                  (lambda ()
                                    (push (float-time) marks)))))
          (project-find-regexp "foobar")
          (cancel-timer timer)
          marks))


which at least suggests that it really is working, and for real
functions which really use call-process.  Since project-find-regexp can
take many seconds on large repos, I think this is useful.

However, my other main test case/use case, being able to paste while
Emacs is in call-process, doesn't work with this change.  Any idea on
how to make that work?