Re: call-process should not block process filters from running

[Eli Zaretskii <eliz@gnu.org>]

> From: sbaugh@catern.com
> Date: Tue, 04 Jul 2023 08:20:39 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> >> 
> >> What is complex about this?
> >
> > Whatever caused you to be surprised that just calling
> > wait_reading_process_output doesn't process selection requests.  How
> > many Lisp programmers understand what is going on in Emacs when such a
> > request is processed, and what that means for their Lisp programs?
> >
> >> What problems can it cause that aren't already covered by "arbitrary
> >> Lisp will run during call-process"?
> >
> > Processing selection requests does not really qualify as "running
> > arbitrary Lisp".  It is much more subtle, and I'm quite sure the
> > mental model of that which most Emacs users have is very simplistic.
> 
> I take your point, but can you be a little more concrete?  I genuinely
> don't know what potential issues you're referring to that can be caused
> by processing selection requests in a new place.  I'm asking just to
> improve my knowledge of Emacs.

I never closely analyzed that, but just follow the control flow of
process_special_events and the functions it calls, and you will see
quite a few non-trivial things.

My point is that most people aren't aware of how this works, so they
will be unable to decide whether their program can or cannot handle
this kind of processing while call-process is in progress.

> > Still not important enough, IMO, for us to have to solve this right
> > away, since you will have the same issue when Emacs is busy with
> > something else, like redrawing the display or even executing some
> > long-running Lisp program.
> 
> Redrawing the display is much quicker

Except when it isn't.  E.g., with long lines, redisplay could take
several hundreds of milliseconds, and sometimes longer.

> and I don't use any long-running Lisp programs.

Maybe you don't, but others might.  "Long" here doesn't have to be
minutes or hours, btw, it could be seconds.  For example, sorting
incoming email could take that long if you've got a lot of it.

> In fact, I move long-running operations out into
> asynchronous subprocesses so they can run concurrently with Emacs.  It's
> only call-process usage that makes Emacs noticeably busy for me.

We are talking about a general-purpose API used a lot in Emacs.  We
are not talking about a feature only for your personal usage -- for
that you can do whatever you want locally.

> (As a tangent, it's kind of sad that this call-process API has ever
> caused Emacs to block/stop running Lisp - subprocesses are inherently
> concurrent!  They are what we use if we want Emacs to *not* block.  I
> understand that that's complex, but it's just kind of ironic.)

There's nothing sad or ironic about that.  call-process has its use
cases in Emacs, and they are valid ones.  Let's not exaggerate.

> OK.  So what's the next step in this step-by-step from your perspective?
> Should we install the option on trunk, let people use it, and gradually
> make what it does broader and more fine-grained over time?

I'd prefer that you run with this locally for some time, and report
back any complications or issues.  You have posted the patch, so if
someone else is interested, they can apply that locally and use it.