Re: wait_reading_process_ouput hangs in certain cases (w/ patches)

[Matthias Dahl <ml_emacs-lists@binary-island.eu>]

[-- Attachment #1: Type: text/plain, Size: 1188 bytes --]

Hello,

after some unexpected / unintended prolonged delay due to personal
reasons, for which I apologize, attached the v2 of my patches.

Basically I have gone a somewhat different route. While working in
some of the requested changes, I noticed that there were still some
pathological cases that were not covered and fixing that would make
things even more convoluted.

So, in this version, the return value is calculated (if necessary)
strategically right before we return from the function call, thus it
cannot be missed and we will always properly signal if data was read
from a wait_proc (either directly or indirectly).

And instead of messing with got_some_output, we exit the loop when we
got some data (directly or indirectly) for our wait_proc if there is
no data to be read for this iteration. This leaves the whole function
logic alone -- except for this key point.

I have addressed the remaining issues, if they still applied. And I
have not been able to trigger a single hang with these patches.

I appreciate any comments and suggestions.

Thanks, again, for all the patience.

So long,
Matthias

-- 
Dipl.-Inf. (FH) Matthias Dahl | Software Engineer | binary-island.eu


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Add-process-output-read-accounting.patch --]
[-- Type: text/x-patch; name="0001-Add-process-output-read-accounting.patch", Size: 1597 bytes --]

From 94e0dc26f45e1a06881b016dd26446c43d339a4d Mon Sep 17 00:00:00 2001
From: Matthias Dahl <matthias.dahl@binary-island.eu>
Date: Tue, 24 Oct 2017 15:55:53 +0200
Subject: [PATCH 1/2] Add process output read accounting

This tracks the bytes read from a process' stdin which is not used
anywhere yet but required for follow-up work.
* src/process.c (read_process_output): Track bytes read from a process.
* src/process.h (struct Lisp_Process): Add nbytes_read to track bytes
read from a process.
---
 src/process.c | 3 +++
 src/process.h | 2 ++
 2 files changed, 5 insertions(+)

diff --git a/src/process.c b/src/process.c
index 2ec10b12ec..17fdf592ec 100644
--- a/src/process.c
+++ b/src/process.c
@@ -5889,6 +5889,9 @@ read_process_output (Lisp_Object proc, int channel)
 	return nbytes;
       coding->mode |= CODING_MODE_LAST_BLOCK;
     }
+  
+  /* Ignore carryover, it's been added by a previous iteration already.  */
+  p->nbytes_read += nbytes;
 
   /* Now set NBYTES how many bytes we must decode.  */
   nbytes += carryover;
diff --git a/src/process.h b/src/process.h
index ab468b18c5..6464a8cc61 100644
--- a/src/process.h
+++ b/src/process.h
@@ -129,6 +129,8 @@ struct Lisp_Process
     pid_t pid;
     /* Descriptor by which we read from this process.  */
     int infd;
+    /* Byte-count modulo (UINTMAX_MAX + 1) for process output read from `infd'.  */
+    uintmax_t nbytes_read;
     /* Descriptor by which we write to this process.  */
     int outfd;
     /* Descriptors that were created for this process and that need
-- 
2.16.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0002-Fix-wait_reading_process_output-wait_proc-hang.patch --]
[-- Type: text/x-patch; name="0002-Fix-wait_reading_process_output-wait_proc-hang.patch", Size: 3307 bytes --]

From b9c05bbfb4559b21deb0ea4e156430dedb60ce41 Mon Sep 17 00:00:00 2001
From: Matthias Dahl <matthias.dahl@binary-island.eu>
Date: Tue, 6 Feb 2018 15:24:15 +0100
Subject: [PATCH 2/2] Fix wait_reading_process_output wait_proc hang

* src/process.c (wait_reading_process_output): If called recursively
through timers and/or process filters via accept-process-output, it is
possible that the output of wait_proc has already been read by one of
those recursive calls, leaving the original call hanging forever if no
further output arrives through that fd and no timeout has been set.
Fix that by using the process read accounting to keep track of how
many bytes have been read and use that as a condition to break out
of the infinite loop and return to the caller as well as to calculate
the proper return value (if a wait_proc is given that is).
---
 src/process.c | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/src/process.c b/src/process.c
index 17fdf592ec..0abbd5fa8e 100644
--- a/src/process.c
+++ b/src/process.c
@@ -5006,6 +5006,7 @@ wait_reading_process_output (intmax_t time_limit, int nsecs, int read_kbd,
   struct timespec got_output_end_time = invalid_timespec ();
   enum { MINIMUM = -1, TIMEOUT, INFINITY } wait;
   int got_some_output = -1;
+  uintmax_t prev_wait_proc_nbytes_read = wait_proc ? wait_proc->nbytes_read : 0;
 #if defined HAVE_GETADDRINFO_A || defined HAVE_GNUTLS
   bool retry_for_async;
 #endif
@@ -5460,6 +5461,8 @@ wait_reading_process_output (intmax_t time_limit, int nsecs, int read_kbd,
       if (nfds == 0)
 	{
           /* Exit the main loop if we've passed the requested timeout,
+             or have read some bytes from our wait_proc (either directly
+             in this call or indirectly through timers / process filters),
              or aren't skipping processes and got some output and
              haven't lowered our timeout due to timers or SIGIO and
              have waited a long amount of time due to repeated
@@ -5467,7 +5470,9 @@ wait_reading_process_output (intmax_t time_limit, int nsecs, int read_kbd,
 	  struct timespec huge_timespec
 	    = make_timespec (TYPE_MAXIMUM (time_t), 2 * TIMESPEC_RESOLUTION);
 	  struct timespec cmp_time = huge_timespec;
-	  if (wait < TIMEOUT)
+	  if (wait < TIMEOUT
+              || (wait_proc
+                  && wait_proc->nbytes_read != prev_wait_proc_nbytes_read))
 	    break;
 	  if (wait == TIMEOUT)
 	    cmp_time = end_time;
@@ -5772,6 +5777,15 @@ wait_reading_process_output (intmax_t time_limit, int nsecs, int read_kbd,
       maybe_quit ();
     }
 
+  /* Timers and/or process filters that we have run could have themselves called
+     `accept-process-output' (and by that indirectly this function), thus
+     possibly reading some (or all) output of wait_proc without us noticing it.
+     This could potentially lead to an endless wait (dealt with earlier in the
+     function) and/or a wrong return value (dealt with here).  */
+  if (wait_proc && wait_proc->nbytes_read != prev_wait_proc_nbytes_read)
+    got_some_output = min (INT_MAX, (wait_proc->nbytes_read
+                                     - prev_wait_proc_nbytes_read));
+
   return got_some_output;
 }
 \f
-- 
2.16.1