Emacs deadlocks on sub-process reads and writes

Emacs deadlocks on sub-process reads and writes

[speech.free]

I am debugging a Clang based code completer (irony-mode). Unfortunately the transfer between Emacs and the Clang sub-process is slow on Windows. We are experiencing frequent freeze-ups due to what looks like a deadlock. The deadlock happens under the following conditions:

1. Emacs reading a large transfer from the sub-process (2+MB, around 2 seconds to finish on Windows after setting w32-pipe-read-delay to 0);

2. While 1. is ongoing Emacs side send a new request to the sub-process. The request is regular source file sized -- it includes the buffer content being edited -- sufficient to cause a block on the write but nothing too big.

When these two conditions are met, Emacs UI reliably locks up until the sub-process is killed. The error message complaining about writing to a dead process after killing the sub-process shows that the Emacs was deadlocked waiting for the write to finish. The process buffer is half filled showing that the process filter function was no longer called. The write to the sub-process, which blocks the UI, can't proceed as it exceeds the size of the small write buffer. The sub-process will not read on its side until it finishes the transfer to Emacs. But for some reason Emacs also stopped reading from the pipe.

After poking around the Emacs code base a little bit it looks to me that when you call process-send-* in lisp Emacs eventually calls send_process in process.c. If the send can't complete it calls wait_reading_process_output, which calls read_and_dispose_of_process_output, which if there is new text received, calls the lisp process-filter. The logic in wait_reading_process_output is too complicated for me to understand if somehow it may decide not to call read_process_output, causing a deadlock. So I am asking for advice here.

A previous related question asked about transfer speed on Windows https://groups.google.com/forum/#!topic/gnu.emacs.help/Y0MYAz8WZUU I believe the slowness is the reason why this manifests itself on Windows but it may also appear on Linux under the right conditions. For a very long thread of discussion on this issue see https://github.com/Sarcasm/irony-mode/issues/131

Please feel free to forward to any Emacs list that may be relevant to solving this issue.

Re: Emacs deadlocks on sub-process reads and writes

[speech.free]

I can now verify that by locking out the blocking write we no longer have the deadlock. So this would appear to be an Emacs bug or feature. I am using the Emacs 24.3.1 (i386-mingw-nt6.2.9200) of 2013-03-17 on MARVIN binary downloaded from Gnu site.

On Saturday, October 18, 2014 6:48:43 PM UTC-7, speec...@gmail.com wrote:
> I am debugging a Clang based code completer (irony-mode). Unfortunately the transfer between Emacs and the Clang sub-process is slow on Windows. We are experiencing frequent freeze-ups due to what looks like a deadlock. The deadlock happens under the following conditions:
> 
> 
> 
> 1. Emacs reading a large transfer from the sub-process (2+MB, around 2 seconds to finish on Windows after setting w32-pipe-read-delay to 0);
> 
> 
> 
> 2. While 1. is ongoing Emacs side send a new request to the sub-process. The request is regular source file sized -- it includes the buffer content being edited -- sufficient to cause a block on the write but nothing too big.
> 
> 
> 
> When these two conditions are met, Emacs UI reliably locks up until the sub-process is killed. The error message complaining about writing to a dead process after killing the sub-process shows that the Emacs was deadlocked waiting for the write to finish. The process buffer is half filled showing that the process filter function was no longer called. The write to the sub-process, which blocks the UI, can't proceed as it exceeds the size of the small write buffer. The sub-process will not read on its side until it finishes the transfer to Emacs. But for some reason Emacs also stopped reading from the pipe.
> 
> 
> 
> After poking around the Emacs code base a little bit it looks to me that when you call process-send-* in lisp Emacs eventually calls send_process in process.c. If the send can't complete it calls wait_reading_process_output, which calls read_and_dispose_of_process_output, which if there is new text received, calls the lisp process-filter. The logic in wait_reading_process_output is too complicated for me to understand if somehow it may decide not to call read_process_output, causing a deadlock. So I am asking for advice here.
> 
> 
> 
> A previous related question asked about transfer speed on Windows https://groups.google.com/forum/#!topic/gnu.emacs.help/Y0MYAz8WZUU I believe the slowness is the reason why this manifests itself on Windows but it may also appear on Linux under the right conditions. For a very long thread of discussion on this issue see https://github.com/Sarcasm/irony-mode/issues/131
> 
> 
> 
> Please feel free to forward to any Emacs list that may be relevant to solving this issue.

Re: Emacs deadlocks on sub-process reads and writes

[Eli Zaretskii]

> Date: Sat, 18 Oct 2014 23:38:15 -0700 (PDT)
> From: speech.free@gmail.com
> 
> I can now verify that by locking out the blocking write we no longer have the deadlock. So this would appear to be an Emacs bug or feature. I am using the Emacs 24.3.1 (i386-mingw-nt6.2.9200) of 2013-03-17 on MARVIN binary downloaded from Gnu site.

Please try the 24.4-rc1 release candidate, this issue is supposed to
be solved there, or at least work better.

If the latest 24.4 version still doesn't solve the problem, please
file a bug report ("M-x report-emacs-bug RET") with all the details.
It would be best if you could show a short self-contained Lisp program
and a small C client program (or an existing utility) that together
demonstrate the problem.

Thanks.

Re: Emacs deadlocks on sub-process reads and writes

[speech.free]

Unfortunately I am not set up to build emacs myself. Is there an rc1 windows binary available somewhere? It is not on the official site yet. It is also not on your ezwinports site. I use msys2/mingw packages. According to a recent thread emacs does no yet build on my platform. My arch linux virtual box does not have X and I use telnet to connect to it. Emacs does not work well over a telnet link on mintty. If official release is just around the corner I can wait for that.