bug#64735: 29.0.92; find invocations are ~15x slower because of ignores

[Dmitry Gutov <dmitry@gutov.dev>]

On 13/09/2023 18:07, Eli Zaretskii wrote:
>> Date: Wed, 13 Sep 2023 17:27:49 +0300
>> Cc: luangruo@yahoo.com, sbaugh@janestreet.com, yantar92@posteo.net,
>>   64735@debbugs.gnu.org
>> From: Dmitry Gutov <dmitry@gutov.dev>
>>
>>>> Does the patch from my last patch-bearing email look similar enough to
>>>> what you're describing?
>>>>
>>>> The one called read_and_insert_process_output.diff
>>>
>>> No, not entirely: it still produces a Lisp string when decoding is
>>> needed, and then inserts that string into the buffer.
>>
>> Are you sure? IIUC the fact that is passes 'curbuf' as the last argument
>> to 'decode_coding_c_string' means that decoding happens inside the
>> buffer. This has been my explanation for the performance improvement anyway.
> 
> Yes, you are right, sorry.

So we're not going to try the gap-based approach? Okay.

>>> Sure, but in this case we don't need any filtering.  It's basically
>>> the same idea as internal-default-process-filter: we just need to
>>> insert the process output into a buffer, and optionally decode it.
>>
>> Pretty much. But that raises the question of what to do with the
>> existing function internal-default-process-filter.
> 
> Nothing.  It will remain as the default filter.

Okay, if you are sure.

>>> We can provide a special filter identified by a symbol.  Such a filter
>>> will not be Lisp-callable, it will exist for the cases where we need
>>> to insert the output into the process buffer.
>>
>> The would be the safest alternative. OTOH, this way we'd pass up on the
>> opportunity to make all existing asynchronous processes without custom
>> filters, a little bit faster in one fell swoop.
> 
> We could change the ones we care about, though.

Which ones do we care about? I've found a bunch of 'make-process' calls 
without :filter specified (flymake backends, ). Do we upgrade them all?

The difference is likely not critical in most of them, but the change 
would likely result in small reduction of GC pressure in the 
corresponding Emacs sessions.

We'll also need to version-guard the ones that are in ELPA.

We don't touch the implementations of functions like start-file-process, 
right?

What about the callers of functions like 
start-file-process-shell-command who want to take advantage of the 
improvement? Are we okay with them all having to call 
(set-process-filter proc 'buffer) on the returned process value?

>> Should we also discuss increasing the default of
>> read-process-output-max? Even increasing it 10x (not necessarily 100x)
>> creates a noticeable difference, especially combined with the proposed
>> change.
> 
> That should be limited to specific cases where we expect to see a lot
> of stuff coming from the subprocess.

So it would be okay to bump it in particular functions? Okay.

> We could also discuss changing
> the default value, but that would require measurements in as many
> cases as we can afford.

If you have some particular scenarios in mind, and what to look out for, 
I could test them out at least on one platform.

I'm not sure what negatives to test for, though. Raising the limit 10x 
is unlikely to lead to an OOM, but I guess some processes could grow 
higher latency?..