Re: call-process should not block process filters from running

[Eli Zaretskii <eliz@gnu.org>]

> From: Spencer Baugh <sbaugh@janestreet.com>
> Date: Sat, 01 Jul 2023 14:24:59 -0400
> 
> >> And AFAIU, such a way does exist.  The implementation of call-process
> >> actually forks a subprocess, then reads from its pipe until EOF, and
> >> then waits in waitpid for the subprocess to exit.  Both the reading
> >> and the waiting are done in a loop.  So one way of making call-process
> >> less blocking is by adding to these loops calls to
> >> wait_reading_process_output, like we do, for example, in sleep-for,
> >> but conditioned by some new variable exposed to Lisp.  Lisp programs
> >> which want this behavior could then activate it by binding that new
> >> variable.  This could give us most or all of the advantages of
> >> non-blocking API without most disadvantages.  (We'd still need to move
> >> to this functionality on a case by case basis, because some of the
> >> considerations against that could still be valid in individual cases.)
> >
> > This sounds great, I would be happy to implement this.  I think we
> > would also want a tiny wrapper in Lisp which binds this new variable
> > then calls call-process, rather than having lots of programs binding
> > the variable directly, to make it easier to change the implementation
> > strategy in the future.
> >
> > I'll work on implementing this new variable.  If you have any other
> > suggestions for it, let me know.
> 
> Just to sanity check before I go down the wrong path: When this variable
> is set, instead of doing the reading from the subprocess's pipe in
> call-process, we'll need to do it in wait_reading_process_output, so
> that other Lisp can run.  We can't just add in calls to
> wait_reading_process_output alongside the existing calls to read,
> because read is blocking, and we need to process other input if there is
> some, even if there's no input from the process under call_process.  A
> similar change will need to happen for waitpid handling.

We read from pipe in chunks, and my idea was to leave the reading code
intact, just call wait_reading_process_output each iteration through
the reading loop.  Long-running call-process calls spend their time in
this loop, reading the process output that takes long time to produce.
So that's where I envision most of the gains will come from.

As for waitpid, I'd initially leave that alone, and only add the
wait_reading_process_output to the loop that protects that from
signals.  IME, by the time we get to waitpid after reading the last
portion of the process output, the process have either already exited
or will do so almost instantly.  If we discover this is not so, we
could later replace waitpid with some wait with timeout in a loop.

I didn't mean to throw away the reading loop and waitpid, because
doing so will require to write a lot of non-trivial C code, and goes
against what I consider to be the main advantage of my idea.