bug#66020: (bug#64735 spin-off): regarding the default for read-process-output-max

[Dmitry Gutov <dmitry@gutov.dev>]

On 21/09/2023 17:59, Eli Zaretskii wrote:
>> Date: Thu, 21 Sep 2023 17:37:23 +0300
>> Cc: stefankangas@gmail.com, monnier@iro.umontreal.ca, 66020@debbugs.gnu.org
>> From: Dmitry Gutov <dmitry@gutov.dev>
>>
>>> How does the throughput depend on this value?  If the dependence curve
>>> plateaus at some lower value, we could use that lower value as a
>>> "good-enough" default.
>>
>> Depends on what we're prepared to call a plateau. Strictly speaking, not
>> really. But we have a "sweet spot": for the process in my original
>> benchmark ('find' with lots of output) it seems to be around 1009600.
>> Here's a table (numbers are different from before because they're
>> results of (benchmark 5 ...) divided by 5, meaning GC is amortized:
>>
>> |    4096 | 0.78 |
>> |   16368 | 0.69 |
>> |   40960 | 0.65 |
>> |  409600 | 0.59 |
>> | 1009600 | 0.56 |
>> | 2009600 | 0.64 |
>> | 4009600 | 0.65 |
> 
> Not enough data points between 40960 and 409600, IMO.  40960 sounds
> like a good spot for the default value.

Or 32K, from the thread linked to previously (datagram size). And ifwe 
were to raise MAX_ALLOCA by 2x, we could still use 'alloca'.

Neither would be optimal for my test scenario, though still an 
improvement. But see my other email with experimental patches, those 
bear improvement with the default 4096.

>>>> And I think we should make the process "remember" the value at its
>>>> creation either way (something touched on in bug#38561): in bug#55737 we
>>>> added an fcntl call to make the larger values take effect. But this call
>>>> is in create_process: so any subsequent increase to a large value of
>>>> this var won't have effect.
>>>
>>> Why would the variable change after create_process?  I'm afraid I
>>> don't understand what issue you are trying to deal with here.
>>
>> Well, what could we lose by saving the value of read-process-output-max
>> in create_process?
> 
> It's already recorded in the size of the pipe, so why would we need to
> record it once more?

'read_process_output' looks it up once more, to set the value of 
'readmax' and allocate char*chars.

Can we get the "recorded" value back from the pipe somehow?