process_send_string blocks?

process_send_string blocks?

[Stephen Leake]

I'm trying to use a background process to parse buffer text, and send
results back to Emacs.

For large buffers, Emacs hangs; only killing the background process
externally recovers. This does not happen for small buffers (< 5k bytes?).

It appears that process-send-string is blocked on a full IO send queue,
while the background process is also blocked on a full IO send queue.

On reading process.c, I believe this should not happen, if the OS supports
EWOULDBLOCK. 

Is that true, even when there is only one call to process-send-string that
sends the entire buffer?

The doc string for process-send-string says "if > 500 chars, string is
sent in bunches", but I see no evidence of that in process.c. Is that
refering to the OS IO queue buffer size?

I'm running on Windows 7, using the Windows binary
emacs-23.4-bin-i386.zip from the FSF FTP site.

Is EWOULDBLOCK supported on this system?

I'll try compiling Emacs from source so I can run the debugger; I've
never tried that on Windows before.

I'd hate to have to code the partial read/partial write logic at the
elisp level.

-- 
-- Stephe

Re: process_send_string blocks?

[Jorgen Schaefer]

On Sat, 06 Sep 2014 03:09:22 -0500
Stephen Leake <stephen_leake@stephe-leake.org> wrote:

> I'm trying to use a background process to parse buffer text, and send
> results back to Emacs.
> 
> For large buffers, Emacs hangs; only killing the background process
> externally recovers. This does not happen for small buffers (< 5k
> bytes?).
> 
> It appears that process-send-string is blocked on a full IO send
> queue, while the background process is also blocked on a full IO send
> queue.
> 
> [...]
>
> I'm running on Windows 7, using the Windows binary
> emacs-23.4-bin-i386.zip from the FSF FTP site.

I have reported a bug that sounds precisely like this here:

http://debbugs.gnu.org/cgi/bugreport.cgi?bug=18396

Eli provided some explanation for the behavior, but says that the
two-way blocking/deadlock shouldn't happen as Emacs uses threads on
Windows for reading.

We're currently trying to see if the problem goes away when using a
background process that does the reading in a separate thread, and thus
not get blocked in the write. The user affected by the bug in my case
seems to have no problems anymore, but as the bug does not happen
reliably in our case, this isn't a full confirmation for a fix yet.

There's some suggestion of using a debugger to figure out what's going
on in that bug report; if you have access to a debugger on windows (Eli
provides an explanation), could you see if you can get more information
from that?

Jorgen

Re: process_send_string blocks?

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Date: Sat, 06 Sep 2014 03:09:22 -0500
> 
> I'm trying to use a background process to parse buffer text, and send
> results back to Emacs.
> 
> For large buffers, Emacs hangs; only killing the background process
> externally recovers. This does not happen for small buffers (< 5k bytes?).
> 
> It appears that process-send-string is blocked on a full IO send queue,
> while the background process is also blocked on a full IO send queue.

Please tell the details of how did you determine this.

> On reading process.c, I believe this should not happen, if the OS supports
> EWOULDBLOCK. 
> 
> Is that true, even when there is only one call to process-send-string that
> sends the entire buffer?

Not sure what are you asking here, since you say you've read the code
and saw the handling of EWOULDBLOCK.

> I'm running on Windows 7, using the Windows binary
> emacs-23.4-bin-i386.zip from the FSF FTP site.
> 
> Is EWOULDBLOCK supported on this system?

Not on writes to pipes, AFAIK.

> I'll try compiling Emacs from source so I can run the debugger; I've
> never tried that on Windows before.

Please do that with the latest pretest, as 24.3 is by now ancient
history.

Also, I suggest to use the latest version 7.8 of GDB, which you can
find here:

  http://sourceforge.net/projects/ezwinports/files/?source=navbar

(Please be sure to read the instructions on the download page, near
its end, which also appear in README.txt, about setting Guile- and
Python-related environment variables.)

To get you up to speed: each subprocess has a dedicated reader thread;
see w32proc.c:reader_thread.  That thread attempts to read 1 byte from
the pipe, and if succeeded, it uses an event object to communicate to
the main (a.k.a. "Lisp") thread that it has read something; sys_select
should then see that and tell the main thread it can read from the
pipe.  It then waits for acknowledge from the main thread before it
attempts to read more.

Writing to the pipe is implemented in w32.c:sys_write (the calls to
'write' elsewhere in Emacs are redirected there).  This runs in the
context of the main thread.

The buffer size for the pipe is defined in w32.c:pipe2.  We pass zero
to the _pipe function, which means "use the default buffer size".  I
couldn't find any information about that default, but you could try
using some specific sizes there (make them integral multiples of 16K)
and see if that makes any difference in your situation.

The first thing I'd do is attach GDB to the hung Emacs and see where
each one of these 2 threads is parked.  (Note that there are more
threads in a running Emacs than just these 2.)

Finally, it is better to continue all this on the bug tracker, perhaps
as part of bug #18396, or a new bug report, if you don't want to mix
up these 2.

Re: process_send_string blocks?

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stephen Leake <stephen_leake@stephe-leake.org>
>> 
>> It appears that process-send-string is blocked on a full IO send queue,
>> while the background process is also blocked on a full IO send queue.
>
> Please tell the details of how did you determine this.

mostly a guess, combined with "there's nothting else wrong".

>> On reading process.c, I believe this should not happen, if the OS supports
>> EWOULDBLOCK. 
>> 
>> Is that true, even when there is only one call to process-send-string that
>> sends the entire buffer?
>
> Not sure what are you asking here, since you say you've read the code
> and saw the handling of EWOULDBLOCK.

Because it is often easy to misunderstand code that I am reading for the
first time. For example, I completely missed the threading stuff you
talk about below.

>> I'm running on Windows 7, using the Windows binary
>> emacs-23.4-bin-i386.zip from the FSF FTP site.
>> 
>> Is EWOULDBLOCK supported on this system?
>
> Not on writes to pipes, AFAIK.

Ah, then that's the problem. Hmm, except below you seem to be saying
maybe it's not?

>> I'll try compiling Emacs from source so I can run the debugger; I've
>> never tried that on Windows before.
>
> Please do that with the latest pretest, as 24.3 is by now ancient
> history.

Ok.

> <snip code description>

thanks for the code info; that will help when I get the debugger
running.

> The first thing I'd do is attach GDB to the hung Emacs and see where
> each one of these 2 threads is parked.  (Note that there are more
> threads in a running Emacs than just these 2.)

I'll give that a try.

> Finally, it is better to continue all this on the bug tracker, perhaps
> as part of bug #18396, or a new bug report, if you don't want to mix
> up these 2.

It's not clear whether this is the same bug; I'll file a new one. I have
a reliable reproducer, which I can easily simplify.

-- 
-- Stephe

Re: process_send_string blocks?

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Date: Sat, 06 Sep 2014 08:11:13 -0500
> 
> >> On reading process.c, I believe this should not happen, if the OS supports
> >> EWOULDBLOCK. 
> >> 
> >> Is that true, even when there is only one call to process-send-string that
> >> sends the entire buffer?
> >
> > Not sure what are you asking here, since you say you've read the code
> > and saw the handling of EWOULDBLOCK.
> 
> Because it is often easy to misunderstand code that I am reading for the
> first time. For example, I completely missed the threading stuff you
> talk about below.

The threads I described are below the level you see in process.c, they
are Windows-specific implementations of 'read', 'write', and 'pselect'
semantics.  On the process.c level, the code handles EWOULDBLOCK on
write, by queuing the rest of the stuff and looping while reading
input.  This is independent of what the lower levels do.

> >> I'm running on Windows 7, using the Windows binary
> >> emacs-23.4-bin-i386.zip from the FSF FTP site.
> >> 
> >> Is EWOULDBLOCK supported on this system?
> >
> > Not on writes to pipes, AFAIK.
> 
> Ah, then that's the problem. Hmm, except below you seem to be saying
> maybe it's not?

I don't know.  It could be.  The implementation of Emacs subprocesses
on Windows is non-trivial, and I cannot claim I have a 100% clear
picture of everything that's going on there.  In addition, there are
other threads that could potentially be involved.

So it's possible that the deadlock is very simple, but there could be
surprises.  Only debugging will tell.

> I have a reliable reproducer, which I can easily simplify.

Please do simplify it and post it.  Having a way of debugging this by
several people will definitely make the process more efficient, and
probably produce a higher quality solution.

Btw, do you see the problem in "emacs -Q"?

Thanks.

Re: process_send_string blocks?

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stephen Leake <stephen_leake@stephe-leake.org>
>>
>> I have a reliable reproducer, which I can easily simplify.
>
> Please do simplify it and post it.  Having a way of debugging this by
> several people will definitely make the process more efficient, and
> probably produce a higher quality solution.

Posted bug#18420. Unfortunately, when I simplify the external process
code, the hang goes away; I don't understand why, yet. 

I'm sure there are bugs in the external code, but I'm having a hard time
reproducing the bugs with purely external testing. So I need Emacs to
stop hanging, so I can find the bugs that cause it to hang :).

In general, nothing the external process does should hang Emacs, short
of crashing the OS.

> Btw, do you see the problem in "emacs -Q"?

Yes, but only with the full external process code and full Ada mode
code.

I'm working on getting the Emacs source compiled and connected to a
debugger. 

I did manage to attach a debugger to the running processes; that's a
first for me :). It was useful for the Ada code (compiled with debug
info), but not for the Emacs process; it says the stack is corrupted.
This is gdb 7.7 from AdaCore, distributed with GNAT GPL 2014, which I
used to build the external process. 

-- 
-- Stephe

Re: process_send_string blocks?

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Date: Sun, 07 Sep 2014 03:21:24 -0500
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Stephen Leake <stephen_leake@stephe-leake.org>
> >>
> >> I have a reliable reproducer, which I can easily simplify.
> >
> > Please do simplify it and post it.  Having a way of debugging this by
> > several people will definitely make the process more efficient, and
> > probably produce a higher quality solution.
> 
> Posted bug#18420. Unfortunately, when I simplify the external process
> code, the hang goes away; I don't understand why, yet. 

Too bad it cannot be simplified.

> In general, nothing the external process does should hang Emacs, short
> of crashing the OS.

Not sure what you are saying here.  How can a crashing OS hang Emacs?

> > Btw, do you see the problem in "emacs -Q"?
> 
> Yes, but only with the full external process code and full Ada mode
> code.

So you are saying that sending text to the subprocess somehow depends
on the major mode?  That'd be surprising.

> I did manage to attach a debugger to the running processes; that's a
> first for me :). It was useful for the Ada code (compiled with debug
> info), but not for the Emacs process; it says the stack is corrupted.

See my response to the bug report: most probably, you are running an
optimized binary, where backtraces blatantly lie.

Re: process_send_string blocks?

[Stephen Leake]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stephen Leake <stephen_leake@stephe-leake.org>
>
>> > Btw, do you see the problem in "emacs -Q"?
>> 
>> Yes, but only with the full external process code and full Ada mode
>> code.
>
> So you are saying that sending text to the subprocess somehow depends
> on the major mode?  That'd be surprising.

No, I just meant "not with the simplified code". So it's not a simple
reproducer. 

In this case, it is the major mode that provides the code that uses the
subprocess, so I can't meaningfully test with other major modes active.

-- 
-- Stephe