bug#64735: 29.0.92; find invocations are ~15x slower because of ignores

[Dmitry Gutov <dmitry@gutov.dev>]

On 13/09/2023 22:32, Eli Zaretskii wrote:

>>> We could change the ones we care about, though.
>>
>> Which ones do we care about? I've found a bunch of 'make-process' calls
>> without :filter specified (flymake backends, ). Do we upgrade them all?
>>
>> The difference is likely not critical in most of them, but the change
>> would likely result in small reduction of GC pressure in the
>> corresponding Emacs sessions.
>>
>> We'll also need to version-guard the ones that are in ELPA.
>>
>> We don't touch the implementations of functions like start-file-process,
>> right?
>>
>> What about the callers of functions like
>> start-file-process-shell-command who want to take advantage of the
>> improvement? Are we okay with them all having to call
>> (set-process-filter proc 'buffer) on the returned process value?
> 
> I think these questions are slightly premature.  We should first have
> the implementation of that filter, and then look for candidates that
> could benefit from it.

The implementation in that patch looks almost complete to me, unless you 
have any further comments. The main difference would be the change in 
the dispatch comparison from

   if (p->filter == Qinternal_default_process_filter)

to

   if (p->filter == Qbuffer)

, I think. Of course I can re-submit the amended patch, if you like.

Regarding documentation, though. How will we describe that new value?

The process filter is described like this in the manual:

    This function gives PROCESS the filter function FILTER.  If FILTER
      is ‘nil’, it gives the process the default filter, which inserts
      the process output into the process buffer.  If FILTER is ‘t’,
      Emacs stops accepting output from the process, unless it’s a
      network server process that listens for incoming connections.

What can we add?

   If FILTER is ‘buffer’, it works like the default one, only a bit faster.

?

> My tendency is to change only callers which
> are in many cases expected to get a lot of stuff from a subprocess, so
> shell buffers are probably out.  But we could discuss that later.

When I'm thinking of start-file-process-shell-command, I have in mind 
project--files-in-directory, which currently uses 
process-file-shell-command. Though I suppose most cases would be more 
easily converted to use make-process (like xref-matches-in-files uses 
process-file for launching a shell pipeline already).

I was also thinking about Flymake backends because those work in the 
background. The outputs are usually small, but can easily grow in rare 
cases, without particular limit. Flymake also runs in the background, 
meaning whatever extra work it has to do (or especially GC pressure), 
affects the delays when editing.

>>> We could also discuss changing
>>> the default value, but that would require measurements in as many
>>> cases as we can afford.
>>
>> If you have some particular scenarios in mind, and what to look out for,
>> I could test them out at least on one platform.
> 
> Didn't think about that enough to have scenarios.
> 
>> I'm not sure what negatives to test for, though. Raising the limit 10x
>> is unlikely to lead to an OOM, but I guess some processes could grow
>> higher latency?..
> 
> With a large buffer and small subprocess output we will ask the OS for
> a large memory increment for no good reason.  Then the following GC
> will want to compact the gap, which means it will be slower.

I wonder what scenario that might become apparent in. Launching many 
small processes at once? Can't think of a realistic test case.

Anyway, if you prefer to put off the discussion about changing the 
default, that's fine by me. Or split into a separate bug.