bug#66020: (bug#64735 spin-off): regarding the default for read-process-output-max

[Dmitry Gutov <dmitry@gutov.dev>]

On 20/09/2023 14:20, Eli Zaretskii wrote:
>> Date: Tue, 19 Sep 2023 22:59:43 +0300
>> Cc: 66020@debbugs.gnu.org
>> From: Dmitry Gutov <dmitry@gutov.dev>
>>
>> - Maybe we declare the difference insignificant and bump the value of
>> read-process-output-max, given that it helps in other cases,
>> - Or try to find out the cause for degradation,
>> - Or keep the default the same, but make it easier to use different
>> value for different processes (meaning, we resurrect the discussion in
>> bug#38561).
> 
> I'd try the same experiment on other use cases, say "M-x grep" and
> "M-x compile" with large outputs, and if you see the same situation
> there (i.e. larger buffers are no worse), try increasing the default
> value on master.

I've run one particular rgrep search a few times (24340 hits, ~44s when 
the variable's value is either 4096 or 409600). And it makes sense that 
there is no difference: compilation modes do a lot more work than just 
capturing the process output or splitting it into strings.

That leaves the question of what new value to use. 409600 is optimal for 
a large-output process but seems too much as default anyway (even if I 
have very little experimental proof for that hesitance: any help with 
that would be very welcome).

I did some more experimenting, though. At a superficial glance, 
allocating the 'chars' buffer at the beginning of read_process_output is 
problematic because we could instead reuse a buffer for the whole 
duration of the process. I tried that (adding a new field to 
Lisp_Process and setting it in make_process), although I had to use a 
value produced by make_uninit_string: apparently simply storing a char* 
field inside a managed structure creates problems for the GC and early 
segfaults. Anyway, the result was slightly _slower_ than the status quo.

So I read what 'alloca' does, and it looks hard to beat. But it's only 
used (as you of course know) when the value is <= MAX_ALLOCA, which is 
currently 16384. Perhaps an optimal default value shouldn't exceed this, 
even if it's hard to create a benchmark that shows a difference. With 
read-process-output-max set to 16384, my original benchmark gets about 
halfway to the optimal number.

And I think we should make the process "remember" the value at its 
creation either way (something touched on in bug#38561): in bug#55737 we 
added an fcntl call to make the larger values take effect. But this call 
is in create_process: so any subsequent increase to a large value of 
this var won't have effect. Might as well remember it there (in a new 
field), then it'll be easier to use different values of it for different 
processes (set using let-binding at the time of the process' creation).