emacs communication with subprocess is slow

emacs communication with subprocess is slow

[Stephen Leake]

This may be related to http://debbugs.gnu.org/cgi/bugreport.cgi?bug=18420

I'm using Emacs pretest 24.3.93, with the patch from 18240, on Windows 7.

I've run into another problem with Emacs communication with external
processes; there seems to be a delay on every task switch. This shows up
both when sending data to the subprocess, and when reading from it.

I haven't tested this on Debian yet.

The delay is not present in Emacs 24.3 on Windows 7.

When sending the contents of an Emacs buffer (source code to be parsed)
to the subprocess, if the read buffer in the subprocess is
large enough to hold the entire contents, no delay is apparent. However,
if the buffer is smaller, so that several reads are needed, then a delay
appears. 

Similarly when sending data from the subprocess to Emacs.

I suspect this doesn't show up with the 'cat' test in 18420 because cat
uses multimegabyte buffers; it's designed for copying gigabyte files.

I can use a large buffer in my subprocess for read as a workaround, but
I have no control over the read buffer in Emacs (perhaps that could be
added?).

Should I reopen 18240, or start a new bug?

-- 
-- Stephe

Re: emacs communication with subprocess is slow

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Date: Fri, 03 Oct 2014 12:51:18 -0500
> 
> This may be related to http://debbugs.gnu.org/cgi/bugreport.cgi?bug=18420
> 
> I'm using Emacs pretest 24.3.93, with the patch from 18240, on Windows 7.
> 
> I've run into another problem with Emacs communication with external
> processes; there seems to be a delay on every task switch. This shows up
> both when sending data to the subprocess, and when reading from it.
> 
> I haven't tested this on Debian yet.
> 
> The delay is not present in Emacs 24.3 on Windows 7.
> 
> When sending the contents of an Emacs buffer (source code to be parsed)
> to the subprocess, if the read buffer in the subprocess is
> large enough to hold the entire contents, no delay is apparent. However,
> if the buffer is smaller, so that several reads are needed, then a delay
> appears. 

Sorry, I don't understand: Emacs 24.3 would simply deaqdlock in this
situation, as we have established in bug #18420.  But you say that not
only does it not hang, it is even faster.  What am I missing?

And what exactly does "large enough" mean here?  What sizes are we
talking about?

Also, can you quantify the delays?

> I can use a large buffer in my subprocess for read as a workaround, but
> I have no control over the read buffer in Emacs (perhaps that could be
> added?).

You could try playing with 2 parameters that currently are fixed: the
size of the pipe buffer (set by pipe2 in w32.c, where we use 0 which
AFAIK defaults to 4KB); and the delay used by send_process in
process.c when it gets EAGAIN/EWOULDBLOCK from the 'write' call
(currently 20 milliseconds).

> Should I reopen 18240, or start a new bug?

A new one, of course.  This is an entirely different issue.

Re: emacs communication with subprocess is slow

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stephen Leake <stephen_leake@stephe-leake.org>
>> Date: Fri, 03 Oct 2014 12:51:18 -0500
>> 
>> This may be related to http://debbugs.gnu.org/cgi/bugreport.cgi?bug=18420
>> 
>> I'm using Emacs pretest 24.3.93, with the patch from 18240, on Windows 7.
>> 
>> I've run into another problem with Emacs communication with external
>> processes; there seems to be a delay on every task switch. This shows up
>> both when sending data to the subprocess, and when reading from it.
>> 
>> I haven't tested this on Debian yet.
>> 
>> The delay is not present in Emacs 24.3 on Windows 7.
>> 
>> When sending the contents of an Emacs buffer (source code to be parsed)
>> to the subprocess, if the read buffer in the subprocess is
>> large enough to hold the entire contents, no delay is apparent. However,
>> if the buffer is smaller, so that several reads are needed, then a delay
>> appears. 
>
> Sorry, I don't understand: Emacs 24.3 would simply deaqdlock in this
> situation, as we have established in bug #18420.  But you say that not
> only does it not hang, it is even faster.  What am I missing?
>
> And what exactly does "large enough" mean here?  What sizes are we
> talking about?
>
> Also, can you quantify the delays?
>
>> I can use a large buffer in my subprocess for read as a workaround, but
>> I have no control over the read buffer in Emacs (perhaps that could be
>> added?).
>
> You could try playing with 2 parameters that currently are fixed: the
> size of the pipe buffer (set by pipe2 in w32.c, where we use 0 which
> AFAIK defaults to 4KB); and the delay used by send_process in
> process.c when it gets EAGAIN/EWOULDBLOCK from the 'write' call
> (currently 20 milliseconds).
>
>> Should I reopen 18240, or start a new bug?
>
> A new one, of course.  This is an entirely different issue.

This sounds like the culprit may be
process-adaptive-read-buffering is a variable defined in `C source code'.
Its value is t

Documentation:
If non-nil, improve receive buffering by delaying after short reads.
On some systems, when Emacs reads the output from a subprocess, the output data
is read in very small blocks, potentially resulting in very poor performance.
This behavior can be remedied to some extent by setting this variable to a
non-nil value, as it will automatically delay reading from such processes, to
allow them to produce more output before Emacs tries to read it.
If the value is t, the delay is reset after each write to the process; any other
non-nil value means that the delay is not reset on write.
The variable takes effect when `start-process' is called.


This variable is a crutch intended to reduce the number of expensive
context switches on single processor systems with a scheduler that ends
up using a pipe as a synchronous context switcher that requires a
context switch for every byte.

It may be worth trying to disable it by default and hope that schedulers
have improved since it was necessary for reasonable performance.

-- 
David Kastrup

Re: emacs communication with subprocess is slow

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

> Sorry, I don't understand: Emacs 24.3 would simply deaqdlock in this
> situation, as we have established in bug #18420.  But you say that not
> only does it not hang, it is even faster.  What am I missing?
>
> And what exactly does "large enough" mean here?  What sizes are we
> talking about?
>
> Also, can you quantify the delays?

I submitted bug 18626; see details there.

>> I can use a large buffer in my subprocess for read as a workaround, but
>> I have no control over the read buffer in Emacs (perhaps that could be
>> added?).
>
> You could try playing with 2 parameters that currently are fixed: the
> size of the pipe buffer (set by pipe2 in w32.c, where we use 0 which
> AFAIK defaults to 4KB); and the delay used by send_process in
> process.c when it gets EAGAIN/EWOULDBLOCK from the 'write' call
> (currently 20 milliseconds).

I'll try this and reply in the bug.

--
-- Stephe