bug#48118: 27.1; 28; Only first process receives output with multiple running processes

[Eli Zaretskii <eliz@gnu.org>]

> Cc: 48118@debbugs.gnu.org
> From: Daniel Mendler <mail@daniel-mendler.de>
> Date: Fri, 30 Apr 2021 18:17:49 +0200
> 
> > Bytes read has a problem when processes produce output a very
> > different rates.  Time spent to handle may (and usually does) mean the
> > filter function does something expensive, it doesn't necessarily tell
> > anything about the output from the subprocess.
> 
> Of course it is not possible to find a perfect scheduling algorithm. But
> how does the OS handle it if you have multiple processes which produce
> output with vastly different rates? I am not claiming this problem has
> been solved, but there are certainly some heuristics. Emacs is also
> dependent on the OS scheduling, depending on how Emacs schedules its
> reads/writes from the processes, the OS scheduler adjusts accordingly.
> This furthermore complicates the picture.

I'm sure patches to tune the scheduling to specific use cases will be
welcome.  My gut feeling is that we will need some variables to allow
Lisp programs to tell Emacs how to handle the various kinds of
processes and combinations thereof, but if you can come up with
patches that automatically adapt to the process's behavior, that would
be even better.

You could also try playing with the value of read-process-output-max,
perhaps enlarging it will make the problem in your case less severe.

> > If you read the code, you will see this isn't what happens.  What
> > happens is that Emacs reads a chunk of output from the first process
> > it sees ready, then it goes back and re-checks which processes are
> > ready -- and in your scenario I think it again sees that the first
> > process is ready.
> 
> This is what we assumed. Emacs could check the second process the next
> time. This way one may get a slightly more fair behavior. It would
> certainly not be perfect and you could throw scenarios at it which would
> make it behave unexpectedly. It may behave a bit more expectedly in the
> common case?

We could try that (conditioned on some new variable) and see if this
has downsides.

> > I suggest to read the code of wait_reading_process_output, it has some
> > non-trivial logic in this department.
> 
> I will do that. Has this problem discussed before?

I don't think so, but my memory is not to be trusted.